Methods and systems for programmatic control of transmitted electronic content

ABSTRACT

Aspects of the invention provide methods, systems, apparatuses and computer program products for implementing a dynamic experimentation service. An example of a method for implementing a dynamic experimentation service includes receiving a set of experiment configuration parameters, determining, by a processor and based on the experiment configuration parameters, an experiment model, the experiment model comprising electronic data indicating an audience selected from a plurality of consumers and a plurality of levels for controlling a communication service in operation to generate an electronic marketing communication for transmission to at least one of the plurality of consumers, receiving a request, from the communication service, the request comprising a communication identifier, determining, based on the communication identifier, at least one of the plurality of levels, and causing the electronic marketing communication to be generated as a treatment comprising the at least one of the plurality of levels.

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/167,436, titled “METHODS AND SYSTEMS FOR PROGRAMMATIC CONTROL OFTRANSMITTED ELECTRONIC CONTENT” filed on May 28, 2015, the entirety ofwhich is incorporated by reference.

TECHNOLOGICAL FIELD

Example embodiments of the present invention relate generally totransmitting electronic marketing communications and, more particularly,to methods, systems, and apparatuses for programmatically controllingthe content of electronic marketing communications and the manner inwhich said electronic marketing communications are transmitted.

BACKGROUND

The applicant has discovered problems with current methods, systems, andapparatuses for evaluating and transmitting electronic marketingcommunications. Through applied effort, ingenuity, and innovation,Applicant has solved many of these identified problems by developing asolution that is embodied by the present invention, which is describedin detail below.

BRIEF SUMMARY

Accordingly, a method, apparatus, and computer program manage thetransmission of electronic marketing communications. Example embodimentsmay include methods, systems, apparatuses, and the like that provide forprogrammatic control of a variety of disparate services that, together,generate one or more electronic marketing communications. Embodimentsmay programmatically implement changes in content, transmissionmechanisms, and other variables associated with generation andtransmission of electronic marketing communications and track a varietyof metrics associated with those electronic marketing communications.These metrics may be analyzed to determine the impact of particularsettings and parameters on the performance of these electronic marketingcommunications. Embodiments may also assist with programmaticdetermination of experiment configuration parameters, including factorand level selection, and audience definition. Some embodiments mayassist with dynamic selection and control of which users are exposed toparticular experimental treatments to improve the quality ofexperimental results and to assist with further programmatic experimentdefinition.

Embodiments include an apparatus for performing audience mapping for adynamic experimentation service. The apparatus includes experimentmanagement circuitry configured to determine a first audience mapping,the first audience mapping defining a first relationship between aplurality of consumers and a plurality of audience groups, determine asecond audience mapping, the second audience mapping defining a secondrelationship between the plurality of consumers and the plurality ofaudience groups, wherein the second relationship is different from thefirst relationship, and store a mapping between the first relationshipand the second relationship, wherein the mapping identifies movement ofthe plurality of consumers across the plurality of audience groups.

The experiment management circuitry may perform the first audiencemapping by a first hashing operation that generates a hash value foreach of the plurality of consumers. The first hashing operation may usea first salt value. The second audience mapping may be performed by asecond hashing operation, and the second hashing operation may use asecond salt value different from the first salt value. The experimentmanagement circuitry may be further configured to log the first saltvalue and the second salt value with the stored mapping. The experimentmanagement circuitry may be further configured to provide the mapping toresult analysis circuitry for use in analysis of a set of experimentresults.

Embodiments also include a method for performing audience mapping for adynamic experimentation service. The method includes determining a firstaudience mapping using a processor, the first audience mapping defininga first relationship between a plurality of consumers and a plurality ofaudience groups, determining a second audience mapping using theprocessor, the second audience mapping defining a second relationshipbetween the plurality of consumers and the plurality of audience groups,wherein the second relationship is different from the firstrelationship, and storing a mapping between the first relationship andthe second relationship, wherein the mapping identifies movement of theplurality of consumers across the plurality of audience groups. Thefirst audience mapping may be performed by a first hashing operationthat generates a hash value for each of the plurality of consumers. Thefirst hashing operation may use a first salt value. The second audiencemapping may be performed by a second hashing operation, and the secondhashing operation may use a second salt value different from the firstsalt value. The method may include logging the first salt value and thesecond salt value with the stored mapping. The method may include usingthe mapping to analyze a set of experiment results.

Embodiments also include a non-transitory computer readable storagemedium comprising instructions that, when executed by a processor, causethe processor to perform audience mapping for a dynamic experimentationservice. The instructions cause the processor to determine a firstaudience mapping, the first audience mapping defining a firstrelationship between a plurality of consumers and a plurality ofaudience groups, determine a second audience mapping, the secondaudience mapping defining a second relationship between the plurality ofconsumers and the plurality of audience groups, wherein the secondrelationship is different from the first relationship, and store amapping between the first relationship and the second relationship,wherein the mapping identifies movement of the plurality of consumersacross the plurality of audience groups.

The first audience mapping may be performed by a first hashing operationthat generates a hash value for each of the plurality of consumers. Thefirst hashing operation may use a first salt value. The second audiencemapping may be performed by a second hashing operation, and the secondhashing operation may use a second salt value different from the firstsalt value. The instructions may further cause the processor to log thefirst salt value and the second salt value with the stored mapping. Theinstructions may further cause the processor to use the mapping toanalyze a set of experiment results.

Embodiments also include yet another apparatus for performing audiencemapping for a dynamic experimentation service. The apparatus includesmeans for determining a first audience mapping using a processor, thefirst audience mapping defining a first relationship between a pluralityof consumers and a plurality of audience groups, means for determining asecond audience mapping using the processor, the second audience mappingdefining a second relationship between the plurality of consumers andthe plurality of audience groups, wherein the second relationship isdifferent from the first relationship, and means for storing a mappingbetween the first relationship and the second relationship, wherein themapping identifies movement of the plurality of consumers across theplurality of audience groups. The first audience mapping may beperformed by a first hashing operation that generates a hash value foreach of the plurality of consumers. The first hashing operation may usea first salt value. The second audience mapping may be performed by asecond hashing operation, and the second hashing operation may use asecond salt value different from the first salt value. The apparatus mayalso include means for logging the first salt value and the second saltvalue with the stored mapping. The apparatus may include means for usingthe mapping to analyze a set of experiment results.

The above summary is provided merely for purposes of summarizing someexample embodiments to provide a basic understanding of some aspects ofthe invention. Accordingly, it will be appreciated that theabove-described embodiments are merely examples and should not beconstrued to narrow the scope or spirit of the invention in any way. Itwill be appreciated that the scope of the invention encompasses manypotential embodiments in addition to those here summarized, some ofwhich will be further described below.

BRIEF DESCRIPTION OF THE DRAWINGS

Having thus described certain example embodiments of the presentdisclosure in general terms, reference will now be made to theaccompanying drawings, which are not necessarily drawn to scale, andwherein:

FIG. 1 illustrates an example system within which embodiments of thepresent invention may operate;

FIG. 2 illustrates a block diagram showing an example device forimplementing a dynamic experimentation system using special-purposecircuitry in accordance with some exemplary embodiments of the presentinvention;

FIG. 3 illustrates an example data flow among a consumer device, aserver, and a merchant device in accordance with some exemplaryembodiments of the present invention;

FIG. 4 illustrates an example of a data flow depicting interactionsbetween services in an exemplary system incorporating multiple servicesin interaction with an experiment service in accordance with someexemplary embodiments of the present invention;

FIG. 5 illustrates an example of a data flow interaction amongcomponents of an experimentation system implementing an experimentsystem in accordance with some exemplary embodiments of the presentinvention;

FIG. 6 illustrates an example of a domain model for an experiment asimplemented by an exemplary experiment service in accordance with someexemplary embodiments of the present invention;

FIG. 7 illustrates a flow diagram depicting an example of a method forperforming an experiment in accordance with some exemplary embodimentsof the present invention;

FIG. 8 illustrates a flow diagram depicting an example of a method foraltering outgoing electronic marketing communications by an experimentservice in accordance with some exemplary embodiments of the presentinvention;

FIG. 9 illustrates a flow diagram depicting an example of a method fortracking results of an experiment in accordance with some exemplaryembodiments of the present invention;

FIG. 10 illustrates a flow diagram depicting an example of a method fordynamically altering an experiment in real-time in accordance with someexemplary embodiments of the present invention; and

FIG. 11 illustrates a flow diagram depicting an example of a method forperforming audience mapping in accordance with some exemplaryembodiments of the present invention.

DETAILED DESCRIPTION Overview

Various embodiments of the present invention are directed to improvedapparatuses, methods, and computer readable media for managing thecontent and transmission of electronic marketing communications toprogrammatically perform experiments, track results, and define newexperiments to test the impact of alterations to electronic marketingcommunications in a manner that provides useful, robust analysisoperating within a flexible framework that does not require tightcoupling between an experiment management service and one or morecommunication services. In this regard, embodiments of the presentinvention provide systems, devices, and frameworks that enablecommunication among multiple services in a multi-service environment tomanage the interactions between those services for the purpose ofgenerating and transmitting electronic marketing communications.Embodiments include tools for defining the parameters of experiments,selecting experimental audiences, and configuring communication servicesto implement the experiments. Embodiments also provide for mapping ofincoming data to particular experimental treatments for the purpose ofdetermining the results of experiments. Embodiments provide for dynamicalteration of experimental parameters based on measured results,including definition of new experiments based on measured results.Embodiments further allow for audience tracking during and acrossexperiments for use in blocking operations.

Definitions

Some embodiments of the present invention will now be described morefully hereinafter with reference to the accompanying drawings, in whichsome, but not all embodiments of the invention are shown. Indeed, theinvention may be embodied in many different forms and should not beconstrued as limited to the embodiments set forth herein; rather, theseembodiments are provided so that this disclosure will satisfy applicablelegal requirements. Like numbers refer to like elements throughout.

As used herein, the terms “data,” “content,” “information,” and similarterms may be used interchangeably to refer to data capable of beingtransmitted, received, and/or stored in accordance with embodiments ofthe present invention. Thus, use of any such terms should not be takento limit the spirit and scope of embodiments of the present invention.Further, where a computing device is described herein to receive datafrom another computing device, it will be appreciated that the data maybe received directly from the another computing device or may bereceived indirectly via one or more intermediary computing devices, suchas, for example, one or more servers, relays, routers, network accesspoints, base stations, hosts, and/or the like, sometimes referred toherein as a “network.” Similarly, where a computing device is describedherein to send data to another computing device, it will be appreciatedthat the data may be sent directly to the another computing device ormay be sent indirectly via one or more intermediary computing devices,such as, for example, one or more servers, relays, routers, networkaccess points, base stations, hosts, and/or the like.

As used herein, the term “promotion and marketing service” may include aservice that is accessible via one or more computing devices and that isoperable to provide promotion and/or marketing services on behalf of oneor more providers that are offering one or more instruments that areredeemable for goods, services, experiences and/or the like. In someexamples, the promotion and marketing service may take the form of aredemption authority, a payment processor, a rewards provider, an entityin a financial network, a promoter, an agent and/or the like. As such,the service is, in some example embodiments, configured to present oneor more promotions via one or more impressions, accept payments forpromotions from consumers, issue instruments upon acceptance of anoffer, participate in redemption, generate rewards, provide a point ofsale device or service, issue payments to providers and/or or otherwiseparticipate in the exchange of goods, services or experiences forcurrency, value and/or the like. The service is also, in some exampleembodiments, configured to offer merchant services such as promotionbuilding (e.g., assisting merchants with selecting parameters for newlycreated promotions), promotion counseling (e.g., offering information tomerchants to assist with using promotions as marketing), promotionanalytics (e.g., offering information to merchants to provide data andanalysis regarding the costs and return-on-investment associated withoffering promotions), and the like.

As used herein, the terms “provider” and “merchant” may be usedinterchangeably and may include, but are not limited to, a businessowner, consigner, shopkeeper, tradesperson, vendor, operator,entrepreneur, agent, dealer, organization or the like that is in thebusiness of a providing a good, service or experience to a consumer,facilitating the provision of a good, service or experience to aconsumer and/or otherwise operating in the stream of commerce. The“provider” or “merchant” need not actually market a product or servicevia the promotion and marketing service, as some merchants or providersmay utilize the promotion and marketing service only for the purpose ofgathering marketing information, demographic information, or the like.

As used herein, the term “consumer” should be understood to refer to arecipient of goods, services, promotions, media, or the like provided bythe promotion and marketing service and/or a merchant. Consumers mayinclude, without limitation, individuals, groups of individuals,corporations, other merchants, and the like.

As used herein, the term “promotion” may include, but is not limited to,any type of offered, presented or otherwise indicated reward, discount,coupon, credit, deal, incentive, discount, media or the like that isindicative of a promotional value or the like that upon purchase oracceptance results in the issuance of an instrument that may be usedtoward at least a portion of the purchase of particular goods, servicesand/or experiences defined by the promotion. Promotions may havedifferent values in different contexts. For example, a promotion mayhave a first value associated with the cost paid by a consumer, known asan “accepted value.” When redeemed, the promotion may be used topurchase a “promotional value” representing the retail price of thegoods. The promotion may also have a “residual value,” reflecting theremaining value of the promotion after expiration. Although consumersmay be primarily focused on the accepted and promotional value of thepromotion, a promotion may also have additional associated values. Forexample, a “cost value” may represent the cost to the merchant to offerthe promotion via the promotion and marketing service, where thepromotion and marketing service receives the cost value for eachpromotion sold to a consumer. The promotion may also include a “returnon investment” value, representing a quantified expected return oninvestment to the merchant for each promotion sold.

For example, consider a promotion offered by the promotion and marketingservice for a $50 meal promotion for $25 at a particular restaurant. Inthis example, $25 would be the accepted value charged to the consumer.The consumer would then be able to redeem the promotion at therestaurant for $50 applied toward their meal check. This $50 would bethe promotional value of the promotion. If the consumer did not use thepromotion before expiration, the consumer might be able to obtain arefund of $22.50, representing a 10% fee to recoup transaction costs forthe merchant and/or promotion and marketing service. This $22.50 wouldbe the residual value of the promotion. If the promotion and marketingservice charged the merchant $3.00 to offer the promotion, the $3.00 feewould be the “cost value.” The “return on investment” value of thepromotion might be dynamically calculated by the promotion and marketingservice based on the expected repeat business generated by the marketingof the promotion, the particular location, the demographics of theconsumer, and the like. For example, the return on investment valuemight be $10.00, reflecting the long term additional profit expected bythe merchant as a result of bringing in a new customer through use of apromotion.

Promotions may be provided to consumers and redeemed via the use of an“instrument.” Instruments may represent and embody the terms of thepromotion from which the instrument resulted. For example, instrumentsmay include, but are not limited to, any type of physical token (e.g.,magnetic strip cards or printed barcodes), virtual account balance(e.g., a promotion being associated with a particular user account on amerchant website), secret code (e.g., a character string that can beentered on a merchant website or point-of-sale), tender, electroniccertificate, medium of exchange, voucher, or the like which may be usedin a transaction for at least a portion of the purchase, acquisition,procurement, consumption or the like of goods, services and/orexperiences as defined by the terms of the promotion.

In some examples, the instrument may take the form of tender that has agiven value that is exchangeable for goods, services and/or experiencesand/or a reduction in a purchase price of a particular good, service orexperience. In some examples, the instrument may have multiple values,such as accepted value, a promotional value and/or a residual value. Forexample, using the aforementioned restaurant as the example provider, anelectronic indication in a mobile application that shows $50 of value tobe used as payment for a meal check at the restaurant. In some examples,the accepted value of the instrument is defined by the value exchangedfor the instrument. In some examples, the promotional value is definedby the promotion from which the instrument resulted and is the value ofthe instrument beyond the accepted value. In some examples, the residualvalue is the value after redemption, the value after the expiry or otherviolation of a redemption parameter, the return or exchange value of theinstrument and/or the like.

As used herein, the term “redemption” refers to the use, exchange orother presentation of an instrument for at least a portion of a good,service or experience as defined by the instrument and its relatedpromotion. In some examples, redemption includes the verification ofvalidity of the instrument. In other example embodiments, redemption mayinclude an indication that a particular instrument has been redeemed andthus no longer retains an actual, promotional and/or residual value(e.g., full redemption). In other example embodiments, redemption mayinclude the redemption of at least a portion of its actual, promotionaland/or residual value (e.g., partial redemption). An example ofredemption, using the aforementioned restaurant as the example provider,is the exchange of the $50 instrument and $50 to settle a $100 mealcheck.

As used herein, the term “impression” refers to a metric for measuringhow frequently consumers are provided with marketing information relatedto a particular good, service, or promotion. Impressions may be measuredin various different manners, including, but not limited to, measuringthe frequency with which content is served to a consumer (e.g., thenumber of times images, websites, or the like are requested byconsumers), measuring the frequency with which electronic marketingcommunications including particular content are sent to consumers (e.g.,a number of e-mails sent to consumers or number of e-mails includingparticular promotion content), measuring the frequency with whichelectronic marketing communications are received by consumers (e.g., anumber of times a particular e-mail is read), or the like. Impressionsmay be provided through various forms of media, including but notlimited to communications, displays, or other perceived indications,such as e-mails, text messages, application alerts, mobile applications,other type of electronic interface or distribution channel and/or thelike, of one or more promotions.

As used herein, the term “electronic marketing information” refers tovarious electronic data and signals that may be interpreted by apromotion and marketing service to provide improved electronic marketingcommunications. Electronic marketing information may include, withoutlimitation, clickstream data (defined below), transaction data (definedbelow), location data (defined below), communication channel data(defined below), discretionary data (defined below), or any other datastored by or received by the promotion and marketing service for use inproviding electronic communications to consumers.

As used herein, the term “clickstream data” refers to electronicinformation indicating content viewed, accessed, edited, or retrieved byconsumers. This information may be electronically processed and analyzedby a promotion and marketing service to improve the quality ofelectronic marketing and commerce transactions offered by, through, andin conjunction with the promotion and marketing service. It should beunderstood that the term “clickstream” is not intended to be limited tomouse clicks. For example, the clickstream data may include variousother consumer interactions, including without limitation, mouse-overevents and durations, the amount of time spent by the consumer viewingparticular content, the rate at which impressions of particular contentresult in sales associated with that content, demographic informationassociated with each particular consumer, data indicating other contentaccessed by the consumer (e.g., browser cookie data), the time or dateon which content was accessed, the frequency of impressions forparticular content, associations between particular consumers orconsumer demographics and particular impressions, and/or the like. Theterm “clickstream event” should be understood to refer to a particularoccurrence, event, or group of events as indicated in the clickstreamdata (e.g., an interaction with a web page, an access of a particularURL, or the like). Accordingly, a stream of clickstream data comprises aplurality of clickstream events.

As used herein, the term “transaction data” refers to electronicinformation indicating that a transaction is occurring or has occurredvia either a merchant or the promotion and marketing service.Transaction data may also include information relating to thetransaction. For example, transaction data may include consumer paymentor billing information, consumer shipping information, items purchasedby the consumer, a merchant rewards account number associated with theconsumer, the type of shipping selected by the consumer for fulfillmentof the transaction, or the like.

As used herein, the term “location data” refers to electronicinformation indicating a particular location. Location data may beassociated with a consumer, a merchant, or any other entity capable ofinteraction with the promotion and marketing service. For example, insome embodiments location data is provided by a location services moduleof a consumer mobile device. In some embodiments, location data may beprovided by a merchant indicating the location of consumers within theirretail location. In some embodiments, location data may be provided bymerchants to indicate the current location of the merchant (e.g., a foodtruck or delivery service). It should be appreciated that location datamay be provided by various systems capable of determining locationinformation, including, but not limited to, global positioning servicereceivers, indoor navigation systems, cellular tower triangulationtechniques, video surveillance systems, or radio frequencyidentification (RFID) location systems.

As used herein, the term “communication channel data” refers toelectronic information relating to the particular device orcommunication channel upon which a merchant or consumer communicateswith the promotion and marketing service. In this regard, communicationchannel data may include the type of device used by the consumer ormerchant (e.g., smart phone, desktop computer, laptop, netbook, tabletcomputer), the Internet Protocol (IP) address of the device, theavailable bandwidth of a connection, login credentials used to accessthe channel (e.g., a user account and/or password for accessing thepromotion and marketing service), or any other data pertaining to thecommunication channel between the promotion and marketing service and anentity external to the promotion and marketing service.

As used herein, the term “discretionary data” refers to electronicinformation provided by a merchant or consumer explicitly to thepromotion and marketing service in support of improved interaction withthe promotion and marketing service. Upon registering with the promotionand marketing service or at any time thereafter, the consumer ormerchant may be invited to provide information that aids the promotionand marketing service in providing services that are targeted to theparticular needs of the consumer or merchant. For example, a consumermay indicate interests, hobbies, their age, gender, or location whencreating a new account. A merchant may indicate the type of goods orservices provided, their retail storefront location, contactinformation, hours of operation, or the like.

It should be appreciated that the term “discretionary data” is intendedto refer to information voluntarily and explicitly provided to thepromotion and marketing service, such as by completing a form or surveyon a website or application hosted by the promotion and marketingservice. However, is should be appreciated that the examples ofdiscretionary data provided above may also be determined implicitly orthrough review or analysis of other electronic marketing informationprovided to the promotion and marketing service. It should also beappreciated that the promotion and marketing service may also gateaccess to certain features or tools based on whether certaindiscretionary data has been provided. For example, the consumer may berequired to provide information relating to their interests or locationduring a registration process.

As used herein, the term “offering parameters” refers to terms andconditions under which the promotion is offered by a promotion andmarketing service to consumers. These offering parameters may includeparameters, bounds, considerations and/or the like that outline orotherwise define the terms, timing, constraints, limitations, rules orthe like under which the promotion is sold, offered, marketed, orotherwise provided to consumers. Example offering parameters include,using the aforementioned restaurant as the example provider, limit oneinstrument per person, total of 100 instruments to be issued, a runduration of when the promotion will be marketed via the promotion andmarketing service, and parameters for identifying consumers to beoffered the promotion (e.g., factors influencing how consumer locationsare used to offer a promotion).

As used herein, the term “redemption parameters” refers to terms andconditions for redeeming or otherwise obtaining the benefit ofpromotions obtained from a promotion and marketing service. Theredemption parameters may include parameters, bounds, considerationsand/or the like that outline the term, timing, constraints, limitations,rules or the like for how and/or when an instrument may be redeemed. Forexample, the redemption parameters may include an indication that theinstrument must be redeemed prior to a specified deadline, for aspecific good, service or experience and/or the like. For example, usingthe aforementioned restaurant as the example provider, the redemptionparameters may specify a limit of one instrument per visit, that thepromotion must be used in store only, or that the promotion must be usedby a certain date.

As used herein, the term “promotion content” refers to display factorsor features that influence how the promotion is displayed to consumers.For example, promotion content may include an image associated with thepromotion, a narrative description of the promotion or the merchant, adisplay template for association with the promotion, or the like.

As used herein, the term “promotion component” is used to refer toelements of a particular promotion that may be selected during apromotion generation process. Promotion components may include anyaspect of a promotion, including but not necessarily limited to offeringparameters, redemption parameters, and promotion content. For example,promotion components may include, but are not limited to, promotiontitles, promotion ledes (e.g., a short text phrase displayed under apromotion title), promotion images, promotion prices, promotion discountlevels, promotion style sheets, promotion fonts, promotion e-mailsubjects, promotion quantities, promotion fine print options, promotionfees assessed to the merchant by the promotion and marketing service, orthe like. Promotion components may also include various flags andsettings associated with registration and verification functions for amerchant offering the promotion, such as whether the identity of themerchant has been verified, whether the merchant is registered with thepromotion and marketing service, or the like.

As used herein, the terms “electronic marketing communication” and“message” refer to any electronically generated information contentprovided by the promotion and marketing service to a consumer for thepurpose of marketing a promotion, good, or service to the consumer.Electronic marketing communications may include any email, short messageservice (SMS) message, web page, application interface, or the likeelectronically generated for the purpose of attempting to sell or raiseawareness of a product, service, promotion, or merchant to the consumer.

It should be appreciated that the term “electronic marketingcommunication” implies and requires some portion of the content of thecommunication to be generated via an electronic process. For example, atelephone call made from an employee of the promotion and marketingservice to a consumer for the purpose of selling a product or servicewould not qualify as an electronic marketing communication, even if theidentity of the call recipient was selected by an electronic process andthe call was dialed electronically, as the content of the telephone callis not generated in an electronic manner. However, a so-called“robo-call” with content programmatically selected, generated, orrecorded via an electronic process and initiated by an electronic systemto notify a consumer of a particular product, service, or promotionwould qualify as an electronic marketing communication. Similarly, amanually drafted e-mail sent from an employee of the promotion andmarketing service to a consumer for the purpose of marketing a productwould not qualify as an electronic marketing communication. However, aprogrammatically generated email including marketing materialsprogrammatically selected based on electronic marketing informationassociated with the recipient would qualify as an electronic marketingcommunication.

As used herein, the term “business analytic data” refers to datagenerated by the promotion and marketing service based on electronicmarketing information to assist with the operation of the promotion andmarketing service and/or one or more merchants. The various streams ofelectronic marketing information provided to and by the promotion andmarketing service allow for the use of sophisticated data analysistechniques that may be employed to identify correlations, relationships,and other associations among elements of electronic marketinginformation. These associations may be processed and formatted by thepromotion and marketing service to provide reports, recommendations, andservices both internal to the promotion and marketing service and tomerchants in order to improve the process by which merchants andpromotion and marketing service engage with consumers. For example, thepromotion and marketing service may analyze the electronic marketinginformation to identify an increased demand for a particular product orservice, and provide an electronic report to a merchant suggesting themerchant offer the particular product or service. Alternatively, thepromotion and marketing service may identify that a particular productor service is not selling or resulting in the merchant losing money,customers, or market share (e.g., after consumers order a particularmenu item, they never come back to the merchant), and suggest that themerchant should discontinue offering that product or service.

It should be appreciated that the term “business analytic data” isintended to refer to electronically and programmatically generated data.For example, a printed report or letter manually drafted by an employeeof the promotion and marketing service would not be said to includebusiness analytic data, even if said data was used by the employeeduring the drafting process, while a data disk or downloaded filecontaining analytics generated by the promotion and marketing servicewould be considered business analytic data.

As used herein, the term “communication service” refers to a particularapplication, module, algorithm, service, device, or the like that iscapable of causing the generation of some portion of an electronicmarketing communication or controlling or configuring the manner inwhich some portion of an electronic marketing communication istransmitted. Communication services may select content for theelectronic marketing communication or message or merely send a signalinstructing another module or service to generate or select content forthe electronic marketing communication. Selection of content of anelectronic marketing communication may include selecting particularproducts, promotions, or the like for inclusion in an electronicmarketing communication, selecting particular non-promotion content(e.g., product shipping notifications, requests for additionaldiscretionary data, user account status data), selecting particularformatting elements (e.g., number of columns, size of images, text fontsize), selecting other content generators for use in generating theelectronic marketing communication (e.g., selection of otherapplications which in turn select content for the electronic marketingcommunication), selection of a title or subject of the electronicmarketing communication, or the like. Communication services may alsoconfigure other aspects of the process of sending the electronicmarketing communication, such as whether and when particular electronicmarketing communications are sent (e.g., by controlling a cadence atwhich an outgoing electronic marketing communications is sent, orselecting a time of day at which the electronic marketing communicationsis sent), selecting which users receive which electronic marketingcommunications, selecting the type of electronic marketing communication(e.g., selecting one or more of an email, “push” notification, or SMSelectronic marketing communication type), or the like.

The term “experiment” is intended to refer to an operation used to testthe impact of one or more parameters used in the generation and/ortransmission of electronic marketing communications. An experiment hasan associated set of experiment configuration parameters. Experimentsmay be defined by those parameters or a stored in a data structure witha unique identifier associated with a group of parameters. Experimentconfiguration parameters may include a set of electronic data sufficientto define a particular experiment. These experiment configurationparameters may include, for example, definitions of the factorsassociated with the experiment, definitions of levels that areassociated with the experiment, definitions of the audience for theexperiment, naming conventions for the experiment and treatmentsgenerated by the experiment, and/or various configuration parameters forthe experiment.

The term “factor” refers to a variable that may be affected by anexperiment. Different communication services may be associated withdifferent factors, and particular services may use different factorswhen participating in the process of generating an electronic marketingcommunication. For example, a service that controls a cadence at whichemails are sent would be associated with an “email cadence” factor,while a service that selects promotion content for inclusion in anelectronic marketing communication would be associated with a “promotioncontent” factor, and a service that selects an email subject for anoutgoing email would be associated with an “email subject” factor. Afactor may have a plurality of “levels” which define specific, mutuallyexclusive values for the factor. For example, using the factor examplesenumerated above, the “email cadence” factor might have levels of“daily, every 4 hours, every 8 hours, weekly”, the “promotion content”factor might have levels of “restaurant promotions, spa promotions,sporting good promotions, vacation getaways”, and the “email subject”might have levels associated with particular text email subjects. Anexample of such a cadence management communication service is describedmore fully in U.S. Provisional Patent Application 62/098,987 and U.S.patent application Ser. Nos. 13/839,102, 13/839,958, 13/838,711, and13/838,452, which are herein incorporated by reference in theirentirety.

It should be appreciated that individual communication services may beassociated with a plurality of factors, each with their own attendantlevels. In some embodiments, levels may also have sub-levels (e.g., fora promotion content selection service, a “promotion content” factormight have a level of “vacation” with attendant sub-levels of“tropical,” “historic,” and “skiing”). It should be appreciated thatselection of factors and levels as described herein necessarily includeselectronic interfaces (e.g., networks, shared memories, applicationprogramming interfaces, or the like) to notify communication services,experiment services, and the like of factor selection, level selection,factor and level availability, and the like.

The term “treatment” is used to refer to a particular group of factorsand levels used to generate an electronic marketing communication orgroup of electronic marketing communications that is/are generatedand/or transmitted according to one or more factors as defined by anexperiment. For example, if a given experiment is testing thesuitability of a particular set of promotion content, electronicmarketing communications sent to test that suitability (e.g., in thecase of an A/B test, a variable group that receives the set of promotioncontent and a control group that does not receive the set of promotioncontent), each electronic marketing communication may represent aseparate treatment as defined by the combination of factor levels usedto generate the electronic marketing communication. In cases where theexperiment is designed to test groupings of electronic marketingcommunications (e.g., where the experiment tests a performance of twotypes of electronic marketing communications sent back-to-back, suchthat factor levels are applied across multiple electronic marketingcommunications), the treatment would be each separate grouping ofelectronic marketing communications. It should be appreciated that insome embodiments a particular electronic marketing communication mayalso function as a treatment for multiple separate experiments, such asin cases where multiple experiments are running simultaneously.Accordingly, descriptions of a given electronic marketing communicationor communications as a “treatment for” a given experiment imply that athe particular electronic marketing communication or communicationsis/are associated with that experiment such that consumer interactionswith that electronic marketing communication(s) provide results for theexperiment.

The term “audience” is understood to refer to a group of consumers orother users that are assigned to receive a treatment or treatments for aparticular experiment. The audience may be selected from a larger groupof users according to various techniques as described herein, anddifferent audiences may be selected for different experiments.Individual users may be assigned to multiple audiences simultaneously.

The term “campaign” is understood to refer to a group of electronicmarketing communications have a particular factor or factors in common.For example, a given campaign may utilize a particular subject line inan email or offer a particular type of discount. It should beappreciated that individual marketing communications may still haveseparate and distinct factors within a campaign. For example, a campaignthat offers a discount percentage may be configured such that allelectronic marketing communications offer a percentage discount, but thevalue of that percentage differs over the campaign (e.g., someelectronic marketing communications offering a 10% discount, some a 20%discount, and others a 25% discount).

Technical Underpinnings and Implementation of Exemplary Embodiments

Merchants, including manufacturers, wholesalers, and retailers, havespent a tremendous amount of time, money, manpower, and other resourcesto determine the best way to market their products to consumers. Whethera given marketing effort is successful is often determined based on thereturn-on-investment offered to the merchant from increased awareness,sales, and the like of the merchant's goods and services in exchange forthe resources spent on the marketing effort. In other words, optimalmarketing techniques generally maximize the benefit to the merchant'sbottom line while minimizing the cost spent on marketing. To this end, amerchant's marketing budget may be spent in a variety of differentmanners including advertising, offering of discounts, conducting marketresearch, and various other known marketing techniques. The end goal ofthese activities is to ensure that products are presented to consumersin a manner that maximizes the likelihood that the consumers willpurchase the product from the merchant that performed the marketingactivities while minimizing the expense of the marketing effort.

The advent of electronic commerce has revolutionized the marketingprocess. While merchants would typically have to perform costly marketresearch such as focus groups, surveys, and the like to obtain detailedinformation on consumer preferences and demographics, the digital agehas provided a wealth of new consumer information that may be used tooptimize the marketing and sales process. As a result, new technologieshave been developed to gather, aggregate, analyze, and reportinformation from a variety of electronic sources.

So-called “clickstream data” provides a robust set of informationdescribing the various interactions consumers have with electronicmarketing communications provided to them by merchants and others.Promotion and marketing services have been developed with sophisticatedtechnology to receive and process this data for the benefit of bothmerchants and consumers. These services assist merchants with marketingtheir products to interested consumers, while reducing the chance that aconsumer will be presented with marketing information in which theconsumer has no interest. Some promotion and marketing services furtherleverage their access to a trove of electronic marketing information toassist merchants and consumers with other tasks, such as offeringimproved merchant point-of-sale systems, improved inventory and supplychain management, improved methods for delivering products and services,and the like.

Unlike conventional marketing techniques related to the use of paper orother physical media (e.g., coupons clipped from a weekly newspaper),promotion and marketing services offer a wealth of additional electronicsolutions to improve the experience for consumers and merchants. Theability to closely monitor user impressions allows the promotion andmarketing service to gather data related to the time, place, and mannerin which the consumer engages with the impression (e.g., viewed,clicked, moused-over) and obtains and redeems the promotion. Thepromotion and marketing service may use this information to determinewhich products and services are most relevant to the consumer'sinterest, and to provide marketing materials related to said productsand services to the consumer, thus improving the quality of theelectronic marketing communications received by the consumer. Merchantsmay be provided with the ability to dynamically monitor and adjust theparameters of promotions offered by the promotion and marketing service,ensuring that the merchant receives a positive return on theirinvestment. For example, the merchant can closely monitor the type,discount level, and quantity sold of a particular promotion on the fly,while with traditional printed coupons the merchant would not be able tomake any changes to the promotion after the coupon has gone to print.Each of these advancements in digital market and promotion distributioninvolve problems unique to the digital environment not before seen intraditional print or television broadcast marketing.

However, these promotion and marketing services are not withoutproblems. Although electronic marketing information provides a wealth ofinformation, the inventors have determined that existing techniques maynot always leverage this information in an efficient or accurate manner.Technology continues to rapidly advance in the field of analytics andthe processing of this information, offering improved data gathering andanalysis techniques, resulting in more relevant and accurate resultsprovided in a more efficient manner. Electronic marketing servicescontinue to evolve and provide improved methods for engaging consumersand spreading awareness of products offered by promotion and marketingservices.

In many cases, the inventors have determined that these services areconstrained by technological obstacles unique to the electronic natureof the services provided, such as constraints on data storage, machinecommunication and processor resources. The inventors have identifiedthat the wealth of electronic marketing information available to theseservices and the robust nature of electronic marketing communicationstechniques present new challenges never contemplated in the world ofpaper coupons and physical marketing techniques. The inventors havefurther determined that even technological methods that leveragecomputers for statistical analysis and consumer behavior modeling (e.g.,television rating systems) fail to address problems associated withproviding relevant, high quality electronic marketing communications(e.g., impressions) to consumers in a manner that maximizes accuracy,minimizes error, is user friendly and provides for efficient allocationof resources. Embodiments of the present invention as described hereinserve to correct these errors and offer improved resource utilization,thus providing improvements to electronic marketing services thataddress problems arising out of the electronic nature of those services.

In particular, the ability to particularly target multiple aspects of anelectronic marketing communication to a particular consumer or group ofconsumers has introduced new technical challenges in the field ofelectronic marketing. A given system for sending electronic marketingcommunications may require interaction among tens or dozens of disparatecommunication services. These services may include separate systems forselecting marketing content (e.g., promotions or products promoted, andattendant text, images, videos, uniform resource locators), formattingselected content (e.g., systems for determining the size and relativepositioning of selected content), selecting the type of electronicmarketing communication (e.g., determining whether to send acommunication as an email or a push notification), schedulingtransmission of the electronic marketing communication (e.g.,determining when a consumer is most likely to be receptive to a givenelectronic marketing communication, managing how frequently electronicmarketing communications are sent to a particular consumer), and more.In order to test the impact of one or more factors affecting theperformance of a given electronic marketing communication, systemadministrators must manually configure each service separately. Eventhen, changes made to one system for a first campaign (e.g., alteringpromotion content) may fail to take into account changes made to anothersystem for a second campaign (e.g., altering transmission cadence), andmanipulation of multiple variables at the same time may complicatederivation of accurate results. For example, because the aforementionedpromotion content service does not notify the transmission cadenceservice that an experiment is running, negative results for electronicmarketing communications altered by the promotion content service may beinterpreted as negative results for alterations made by the transmissioncadence service, or vice-versa.

Recognizing these and other problems, the inventors have recognized aneed for methods, systems, and apparatuses that provide a robustframework for enabling intra-service communication and management ofexperiments to improve the process by which electronic marketingcommunications are generated. To this end, the inventors have developedservices, systems, and applications that are capable of accepting a setof experiment configuration parameters. Embodiments use these experimentconfiguration parameters to identify particular factors and levels formanipulation to generate a set of experiment treatments that result intesting of the impact of the selected factors and levels. Embodimentsfunction to programmatically notify particular communication services tocause those communication services to implement the selected factors andlevels. Embodiments further provide a framework for establishingstatistically significant sample sizes, controlling the assignedaudience, mapping a set of clickstream data to particular experimenttreatments, and providing real-time programmatic feedback to dynamicallyalter the experiment. Embodiments also provide for tools to assist withblocking of an experiment audience to improve the accuracy of results.

System Architecture

Methods, apparatuses, and computer program products of the presentinvention may be embodied by any of a variety of devices. For example,the method, apparatus, and computer program product of an exampleembodiment may be embodied by a networked device, such as a server orother network entity, configured to communicate with one or moredevices, such as one or more client devices. Additionally oralternatively, the computing device may include fixed computing devices,such as a personal computer or a computer workstation. Still further,example embodiments may be embodied by any of a variety of mobileterminals, such as a portable digital assistant (PDA), mobile telephone,smartphone, laptop computer, tablet computer, or any combination of theaforementioned devices.

In this regard, FIG. 1 discloses an example computing system 100 withinwhich embodiments of the present invention may operate. Merchants mayaccess a promotion and marketing service 102 via a network 112 (e.g.,the Internet, or the like) using computer devices 108A through 108N and110A through 110N, respectively (e.g., one or more consumer devices108A-108N or one or more merchant devices 110A-110N). The promotion andmarketing service 102 may function to manage transmission of electronicmarketing communications as described herein and below. Moreover, thepromotion and marketing service 102 may comprise a server 104 incommunication with a database 106.

The server 104 may be embodied as a computer or computers as known inthe art. The server 104 may provide for receiving of electronic datafrom various sources, including but not necessarily limited to theconsumer devices 108A-108N and the merchant devices 110A-110N. Forexample, the server 104 may be operable to receive and processclickstream data provided by the consumer devices 108 and/or themerchant devices 110. The server 104 may also facilitate e-commercetransactions based on transaction data provided by the consumer devices108 and/or the merchant devices 110. The server 104 may facilitate thegeneration and providing of various electronic marketing communicationsand marketing materials based on the received electronic marketinginformation.

The database 106 may be embodied as a data storage device such as aNetwork Attached Storage (NAS) device or devices, or as a separatedatabase server or servers. The database 106 includes informationaccessed and stored by the server 104 to facilitate the operations ofthe promotion and marketing service 102. For example, the database 106may include, without limitation, user account credentials for systemadministrators, merchants, and consumers, data indicating the productsand promotions offered by the promotion and marketing service,electronic marketing information, analytics, reports, financial data,and/or the like.

The consumer devices 108A-108N may be any computing device as known inthe art and operated by a consumer. Electronic data received by theserver 104 from the consumer devices 108A-108N may be provided invarious forms and via various methods. For example, the consumer devices108A-108N may include desktop computers, laptop computers, smartphones,netbooks, tablet computers, wearables, and the like. The information maybe provided through various sources on these consumer devices.

In embodiments where a consumer device 108 is a mobile device, such as asmart phone or tablet, the consumer device 108 may execute an “app” tointeract with the promotion and marketing service 102. Such apps aretypically designed to execute on mobile devices, such as tablets orsmartphones. For example, an app may be provided that executes on mobiledevice operating systems such as Apple Inc.'s iOS®, Google Inc.'sAndroid®, or Microsoft Inc.'s Windows 8®. These platforms typicallyprovide frameworks that allow apps to communicate with one another andwith particular hardware and software components of mobile devices. Forexample, the mobile operating systems named above each provideframeworks for interacting with location services circuitry, wired andwireless network interfaces, user contacts, and other applications in amanner that allows for improved interactions between apps while alsopreserving the privacy and security of consumers. In some embodiments, amobile operating system may also provide for improved communicationinterfaces for interacting with external devices (e.g., home automationsystems, indoor navigation systems, and the like). Communication withhardware and software modules executing outside of the app is typicallyprovided via application programming interfaces (APIs) provided by themobile device operating system.

The promotion and marketing service 102 may leverage the applicationframework offered by the mobile operating system to allow consumers todesignate which information is provided to the app and which may then beprovided to the promotion and marketing service 102. In someembodiments, consumers may “opt in” to provide particular data to thepromotion and marketing service 102 in exchange for a benefit, such asimproved relevancy of marketing communications offered to the user. Insome embodiments, the consumer may be provided with privacy informationand other terms and conditions related to the information provided tothe promotion and marketing service 102 during installation or use ofthe app. Once the consumer provides access to a particular feature ofthe mobile device, information derived from that feature may be providedto the promotion and marketing service 102 to improve the quality of theconsumer's interactions with the promotion and marketing service.

For example, the consumer may indicate that they wish to providelocation information to the app from location services circuitryincluded in their mobile device. Providing this information to thepromotion and marketing service 102 may enable the promotion andmarketing service 102 to offer promotions to the consumer that arerelevant to the particular location of the consumer (e.g., by providingpromotions for merchants proximate to the consumer's current location).It should be appreciated that the various mobile device operatingsystems may provide the ability to regulate the information provided tothe app associated with the promotion and marketing service 102. Forexample, the consumer may decide at a later point to disable the abilityof the app to access the location services circuitry, thus limiting theaccess of the consumer's location information to the promotion andmarketing service 102.

Various other types of information may also be provided in conjunctionwith an app executing on the consumer's mobile device. For example, ifthe mobile device includes a social networking feature, the consumer mayenable the app to provide updates to the consumer's social network tonotify friends of a particularly interesting promotion. It should beappreciated that the use of mobile technology and associated appframeworks may provide for particularly unique and beneficial uses ofthe promotion and marketing service through leveraging the functionalityoffered by the various mobile operating systems.

Additionally or alternatively, the consumer device 108 may interactthrough the promotion and marketing service 102 via a web browser. Asyet another example, the consumer device 108 may include varioushardware or firmware designed to interface with the promotion andmarketing service 102 (e.g., where the consumer device 108 is apurpose-built device offered for the primary purpose of communicatingwith the promotion and marketing service 102, such as a store kiosk).

The merchant devices 110A-110N may be any computing device as known inthe art and operated by a merchant. For example, the merchant devices110A-110N may include a merchant point-of-sale, a merchant localmarketing device, a merchant e-commerce server, a merchant inventorysystem, or a computing device accessing a web site designed to providemerchant access (e.g., by accessing a web page via a browser using a setof merchant account credentials). The merchant devices 110A-110N mayalso be mobile devices as described above with respect to the consumerdevices 108A-108N.

Electronic data received by the promotion and marketing service 102 fromthe merchant devices 110A-110N may also be provided in various forms andvia various methods. For example, the merchant devices 110A-110N mayprovide real-time transaction and/or inventory information as purchasesare made from the merchant. In other embodiments, the merchant devices110A-110N may be employed to provide information to the promotion andmarketing service 102 to enable the promotion and marketing service 102to generate promotions or other marketing information to be provided toconsumers.

An example of a data flow for exchanging electronic marketinginformation among one or more consumer devices, merchant devices, andthe promotion and marketing service is described below with respect toFIG. 3.

Example Apparatuses for Implementing Embodiments of the PresentInvention

The server 104 may be embodied by one or more computing systems, such asapparatus 200 shown in FIG. 2. As illustrated in FIG. 2, the apparatus200 may include a processor 202, a memory 204, input/output circuitry206, communications circuitry 208, experiment configuration circuitry210, experiment management circuitry 212, result analysis circuitry 214,and communication service circuitry 216. The apparatus 200 may beconfigured to execute the operations described above with respect toFIG. 1 and below with respect to FIGS. 4-11. Although these components202-216 are described with respect to functional limitations, it shouldbe understood that the particular implementations necessarily includethe use of particular hardware. It should also be understood thatcertain of these components 202-216 may include similar or commonhardware. For example, two sets of circuitry may both leverage use ofthe same processor, network interface, storage medium, or the like toperform their associated functions, such that duplicate hardware is notrequired for each set of circuitry. The use of the term “circuitry” asused herein with respect to components of the apparatus should thereforebe understood to include particular hardware configured to perform thefunctions associated with the particular circuitry as described herein.

The term “circuitry” should be understood broadly to include hardwareand, in some embodiments, software for configuring the hardware. Forexample, in some embodiments, “circuitry” may include processingcircuitry, storage media, network interfaces, input/output devices, andthe like. In some embodiments, other elements of the apparatus 200 mayprovide or supplement the functionality of particular circuitry. Forexample, the processor 202 may provide processing functionality, thememory 204 may provide storage functionality, the communicationscircuitry 208 may provide network interface functionality, and the like.

In some embodiments, the processor 202 (and/or co-processor or any otherprocessing circuitry assisting or otherwise associated with theprocessor) may be in communication with the memory 204 via a bus forpassing information among components of the apparatus. The memory 204may be non-transitory and may include, for example, one or more volatileand/or non-volatile memories. In other words, for example, the memorymay be an electronic storage device (e.g., a computer readable storagemedium). The memory 204 may be configured to store information, data,content, applications, instructions, or the like, for enabling theapparatus to carry out various functions in accordance with exampleembodiments of the present invention.

The processor 202 may be embodied in a number of different ways and may,for example, include one or more processing devices configured toperform independently. Additionally or alternatively, the processor mayinclude one or more processors configured in tandem via a bus to enableindependent execution of instructions, pipelining, and/ormultithreading. The use of the term “processing circuitry” may beunderstood to include a single core processor, a multi-core processor,multiple processors internal to the apparatus, and/or remote or “cloud”processors.

In an example embodiment, the processor 202 may be configured to executeinstructions stored in the memory 204 or otherwise accessible to theprocessor. Alternatively or additionally, the processor may beconfigured to execute hard-coded functionality. As such, whetherconfigured by hardware or software methods, or by a combination thereof,the processor may represent an entity (e.g., physically embodied incircuitry) capable of performing operations according to an embodimentof the present invention while configured accordingly. Alternatively, asanother example, when the processor is embodied as an executor ofsoftware instructions, the instructions may specifically configure theprocessor to perform the algorithms and/or operations described hereinwhen the instructions are executed.

In some embodiments, the apparatus 200 may include input/outputcircuitry 206 that may, in turn, be in communication with processor 202to provide output to the user and, in some embodiments, to receive anindication of a user input. The input/output circuitry 206 may comprisea user interface and may include a display and may comprise a web userinterface, a mobile application, a client device, a kiosk, or the like.In some embodiments, the input/output circuitry 206 may also include akeyboard, a mouse, a joystick, a touch screen, touch areas, soft keys, amicrophone, a speaker, or other input/output mechanisms. The processorand/or user interface circuitry comprising the processor may beconfigured to control one or more functions of one or more userinterface elements through computer program instructions (e.g., softwareand/or firmware) stored on a memory accessible to the processor (e.g.,memory 204, and/or the like).

The communications circuitry 208 may be any means such as a device orcircuitry embodied in either hardware or a combination of hardware andsoftware that is configured to receive and/or transmit data from/to anetwork and/or any other device, circuitry, or module in communicationwith the apparatus 200. In this regard, the communications circuitry 208may include, for example, a network interface for enablingcommunications with a wired or wireless communication network. Forexample, the communications circuitry 208 may include one or morenetwork interface cards, antennae, buses, switches, routers, modems, andsupporting hardware and/or software, or any other device suitable forenabling communications via a network. Additionally or alternatively,the communication interface may include the circuitry for interactingwith the antenna(s) to cause transmission of signals via the antenna(s)or to handle receipt of signals received via the antenna(s).

The experiment configuration circuitry 210 includes hardware configuredto receive input to configure one or more experiments, and to processthat input into a format suitable for use by the experiment managementcircuitry 212. The experiment configuration circuitry 210 may beconfigured to execute an application or applications suitable forreceiving experiment configuration parameters and to generate one ormore data structures corresponding to an experiment having the receivedparameters. For example, the experiment configuration circuitry 210 mayinclude a graphical user interface allowing a user to select particularfactors, levels, audience characteristics, and the like to define anexperiment or experiments. The experiment configuration circuitry 210may also include hardware and software configured to programmaticallydefine experiments, such as in response to receiving particular resultsfrom a previous experiment. The experiment configuration circuitry 210may include an interface populated with factors and levels dynamicallyreceived from one or more communication services executed or managed bythe communication service circuitry 216. For example, uponinitialization of an experiment service or communication service, thecommunication service may report the factors and/or levels that thecommunication service is configured to receive for use in experiments,and the reported factors and levels may be included in a graphical userinterface provided by the experiment configuration circuitry 210, suchthat a user may select those factors and levels via the user interface.The experiment configuration circuitry 210 may include a processor, suchas the processor 202, to execute one or more applications to provide thefunctionality of the experiment configuration circuitry 210. Theexperiment configuration circuitry 210 may provide a graphical userinterface via the input/output circuitry 206 and/or the communicationscircuitry 208, such as on a local display device (e.g., via theinput/output circuitry 206) or by a hosted web application (e.g., viathe communications circuitry 208).

The experiment management circuitry 212 may include hardware configuredto initiate, monitor, control, and manage experiments as created by theexperiment configuration circuitry 210. The experiment managementcircuitry 212 may operate to communicate with one or more communicationservices to notify those communication services what particular factorsand levels should be employed based on a set of experiment definitions.In some embodiments, the experiment management circuitry 212 mayimplement a communications mechanism with the communication servicessuch that the communication service calls an Application ProgrammingInterface (API) function when generating an electronic marketingcommunication to notify the experiment management circuitry 212 and toreceive one or more levels for one or more factors implemented by thecommunications service in response to the API function call. Theexperiment management circuitry 212 may provide one or more callbackfunctions, applications, processes, or the like for managing these andother API calls. The available factors/levels for the communicationservice(s) may be stored as communication service capability data storedin a memory, such as the memory 204.

The experiment management circuitry 212 may also be configured to manageexperiment audiences, including determining which consumers areassociated with which experiment at any given time. The experimentmanagement circuitry 212 may be configured to manage a plurality ofexperiments at any given time. The experiment management circuitry 212may also be configured to store and manage experiments defined by theexperiment configuration circuitry 210. The experiments may be stored asa set of experiment model data stored in a memory, such as the memory204. To perform these functions and various other functions asenumerated herein, the experiment management circuitry 212 may include aprocessor, such as the processor 202. Embodiments of exemplaryimplementations of these features and other features of the experimentmanagement circuitry 212 are described further below with respect toFIGS. 3-11.

The result analysis circuitry 214 may include hardware configured toreceive electronic data, such as clickstream data, associated with oneor more electronic marketing communications. These electronic marketingcommunications may have been generated according to one or moreexperiments managed by the experiment management circuitry 212, and theresult analysis circuitry 214 may provide a mapping of receivedclickstream data to particular treatments, consumers, and/orexperiments. In some embodiments, the result analysis circuitry 214 usesan attribution model to attribute particular logged and tracked eventsto particular consumers, treatments, and/or experiments. The resultanalysis circuitry 214 may also be configured to derive conclusionsbased on the received clickstream data and a mapping to particulartreatments and/or levels associated with the experiment. For example,the result analysis circuitry may receive an indication of whichconsumers were associated with successful electronic marketingcommunications, which consumers were associated with unsuccessfulelectronic marketing communications, and, of those consumers, which wereassociated with particular treatments and/or levels as defined by anexperiment. The result analysis circuitry 214 may use this data toidentify a correlation between the success of a particular treatment andparticular levels of factors used in that treatment.

The term “success” in the context of determining whether a particulartreatment or electronic marketing communication was “successful” may bedefined in a variety of ways. For example, “success” may mean that aconsumer clicked at least one promotion associated with the treatment,that the consumer purchased a product or service associated with thetreatment, that the consumer shared a particular product or serviceassociated with the treatment via a social media website, or any otherfactor that may be deemed worthy of tracking and analysis. It should beappreciated that different experiments may have different “success”criteria, and such criteria may be dynamically defined, such as byinteraction with the experiment configuration circuitry 210. Theexperiment configuration circuitry 214 may include a processor, such asthe processor 202, to perform these functions. Clickstream data andvarious other data used for measurement of results and attribution ofdata to a particular treatment/experiment may be received for acommunication interface, such as the communications circuitry 208.

The communication service circuitry 216 includes hardware configured toparticipate in a process for generating and transmitting one or moreelectronic marketing communications. In this regard, the communicationservice circuitry 216 may implement the functionality of a plurality ofseparate services and/or applications. For example, communicationservices implemented by the communication service circuitry 216 mayinclude, without limitation, services for selecting content forelectronic marketing communications (e.g., selecting particularpromotions or products to be advertised, selecting types of promotionsor products, selecting content generator applications that in turnprovide content), services for determining whether and when to sendelectronic marketing communications (e.g., services that determine whattime of day, day of week, or month of year to send electronic marketingcommunications, or services that implement criteria-based transmissionof electronic marketing communications), services that manage atransmission frequency of an electronic marketing communication (e.g., aservice that ensures that consumers do not receive electronic marketingcommunications with greater than a particular frequency), or the like.The communication service circuitry 216 may communicate with theexperiment management circuitry 212 to notify the experiment managementcircuitry 212 of which factors and/or levels are available to controlthe particular communication services implemented by the communicationservice circuitry 216. The communication service circuitry 216 maysubsequently receive instructions from the experiment managementcircuitry 212 to adjust levels for outgoing electronic marketingcommunications for use in experiment treatments. This and otherfunctionality of the communication service circuitry 216 may beimplemented by one or more applications executing on a processor, suchas the processor 202, or a communication interface, such as thecommunications circuitry 208. It should also be appreciated that, insome embodiments, one or more communication services are implemented oncomputing nodes located remotely to the apparatus 200, and thecommunication service circuitry 216 of the apparatus may includehardware configured to provide a remote interface (e.g., an APIinterface) for communicating with those remotely executingcommunications services.

As will be appreciated, any such computer program instructions and/orother type of code may be loaded onto a computer, processor or otherprogrammable apparatus's circuitry to produce a machine, such that thecomputer, processor other programmable circuitry that execute the codeon the machine create the means for implementing various functions,including those described herein.

It is also noted that all or some of the information presented byexample displays described herein can be based on data that is received,generated and/or maintained by one or more components of apparatus 200.In some embodiments, one or more external systems (such as a remotecloud computing and/or data storage system) may also be leveraged toprovide at least some of the functionality discussed herein.

As described above and as will be appreciated based on this disclosure,embodiments of the present invention may be configured as methods,mobile devices, backend network devices, and the like. Accordingly,embodiments may comprise various means including entirely of hardware orany combination of software and hardware. Furthermore, embodiments maytake the form of a computer program product on at least onenon-transitory computer-readable storage medium having computer-readableprogram instructions (e.g., computer software) embodied in the storagemedium. Any suitable computer-readable storage medium may be utilizedincluding non-transitory hard disks, CD-ROMs, flash memory, opticalstorage devices, or magnetic storage devices.

Example Electronic Marketing Information Service Data Flow

FIG. 3 depicts an example data flow 300 illustrating interactionsbetween a server 302, one or more consumer devices 304, and one or moremerchant devices 306. The server 302 may be implemented in the same or asimilar fashion as the server 104 as described above with respect toFIG. 1 and/or the apparatus 200 described above with respect to FIG. 2,the one or more consumer devices 304 may be implemented in the same or asimilar fashion as the consumer devices 108A-108N as described abovewith respect to FIG. 1, and the one or more merchant devices 306 may beimplemented in the same or a similar fashion as the merchant devices110A-110N as described above with respect to FIG. 1.

The data flow 300 illustrates how electronic information may be passedamong various systems when employing a server 302 in accordance withembodiments of the present invention. The one or more consumer devices304 and/or one or more merchant devices 306 may provide a variety ofelectronic marketing information to the server 302 for use in providingpromotion and marketing services to the consumer. This electronicmarketing information may include, but is not limited to, location data,clickstream data, transaction data, communication channel data, and/ordiscretionary data.

As a result of transactions performed between the one or more consumerdevices 304 and the server 302, the server 302 may provide fulfillmentdata to the consumer devices. The fulfillment data may includeinformation indicating whether the transaction was successful, thelocation and time the product will be provided to the consumer,instruments for redeeming promotions purchased by the consumer, or thelike.

In addition to the e-commerce interactions with the one or more consumerdevices 304 offered by the server 302, the server 302 may leverageinformation provided by the consumer devices to improve the relevancy ofelectronic marketing communications to individual consumers or groups ofconsumers. In this manner, the server 302 may determine promotions,goods, and services that are more likely to be of interest to aparticular consumer or group of consumers based on clickstream data,location data, and other information provided by and/or relating toparticular consumers. For example, the server 302 may detect thelocation of a consumer based on location data provided by the consumerdevice, and offer promotions based on the proximity of the consumer tothe merchant associated with those promotions.

Alternatively, the server 302 may note that the consumer has an interestin a particular hobby (e.g., skiing) based on electronic marketinginformation associated with the consumer (e.g., a browser cookie thatindicates they frequently visit websites that provide snowfall forecastsfor particular ski resorts), and offer promotions associated with thathobby (e.g., a promotion offering discounted ski equipment rentals orlift tickets). It should be appreciated that a variety of differenttypes of electronic marketing information could be provided to theserver 302 for the purpose of improving the relevancy of marketingcommunications. It should also be appreciated that this electronicmarketing information may be received from a variety of electronicsources, including various consumer devices, merchant devices, and othersources both internal and external to a promotion and marketing service.For example, other data sources may include imported contact databasesmaintained by merchants, electronic survey questions answered byconsumers, and/or various other forms of electronic data.

It should also be appreciated that the server 302 may also control otherfactors of the electronic marketing communications sent to the consumerother than the particular promotions included in the electronicmarketing communication. For example, the server 302 may determine theform, structure, frequency, and type of the electronic marketingcommunication. As with the content of the electronic marketingcommunication, these factors may be programmatically determinedaccording to various methods, factors, and processes based on electronicdata received by the server 302 for the purpose of maximize thelikelihood that the communication will be relevant to the recipientconsumer.

The server 302 interactions with the one or more merchant devices 306may be related to enabling the merchant to market their products using apromotion and marketing service. For example, the one or more merchantdevices 306 may provide promotion data defining one or more promotionsto be offered by the promotion and marketing service on behalf of themerchant. The server 302 may receive this information and generateinformation for providing such promotions via an e-commerce interface,making the promotions available for purchase by consumers. The server302 may also receive information about products from the one or moremerchant devices 306. For example, a merchant may provide electronicmarketing information indicating particular products, product prices,inventory levels, and the like to be marketed via a promotion andmarketing service. The server 302 may receive this information andgenerate listing information to offer the indicating products toconsumers via a promotion and marketing service.

The one or more merchant devices 306 may also receive information fromthe server 302. For example, in some embodiments a merchant may obtainaccess to certain business analytic data aggregated, generated, ormaintained by the server 302. As a particular example, a merchant mightoffer to pay for consumer demographic data related to products orservices offered by the merchant. It should be appreciated however, thata merchant may not need to list any products or services via thepromotion and marketing service in order to obtain such data. Forexample, the promotion and marketing service may enable merchants toaccess electronic marketing data offered via the promotion and marketingservice based on a subscription model.

The one or more merchant devices 306 may also receive electroniccompensation data from the server 302. For example, when a promotion orproduct is sold by the promotion and marketing service on behalf of themerchant, a portion of the received funds may be transmitted to themerchant. The compensation data may include information sufficient tonotify the merchant that such funds are being or have been transmitted.In some embodiments, the compensation data may take the form of anelectronic wire transfer directly to a merchant account. In some otherembodiments, the compensation data may indicate that a promotion orproduct has been purchased, but the actual transfer of funds may occurat a later time. For example, in some embodiments, compensation dataindicating the sale of a promotion may be provided immediately, butfunds may not be transferred to the merchant until the promotion isredeemed by the consumer.

Embodiments advantageously provide for improvements to the server 302 byoffering a flexible, robust framework for implementing experimentswithout requiring direct control or intervention to reconfigureindividual communication services. Embodiments also provide forprogrammatic configuration of experiment configuration parameters andmapping of clickstream data to particular electronic marketingcommunications and experiment treatments for the purpose of generatingexperiment results. These improvements serve to reduce the number ofelectronic marketing communications that must be sent by the server 302in order to produce sales, conserving system resources.

Exemplary Interactions Between Services

FIG. 4 illustrates an example data flow interaction 400 among aplurality of services implementing embodiments of the present invention.An experiment configuration client 402 may be employed to define a setof experiment configuration parameters for an experiment to beimplemented by an experiment service 404. The experiment configurationclient 402 may be similarly configured to and/or implemented by theexperiment configuration circuitry as described above. The experimentservice 404 may be similarly configured and/or implemented by theexperiment management circuitry as described above.

The experiment service 404 may implement an API that allowscommunication with one or more experiment clients 408. The experimentclients 408 may be functionality implemented alongside of or as part ofone or more participating services 410. The participating services 410may be communication services as described above. In operation, theparticipating services 410 may employ the experiment clients 408 toaccess or register with the experiment service 404. When a givenparticipating service 410 is initialized, the experiment client 408associated with the participating service 410 may notify the experimentservice 404 of the availability of the participating service 410 to beconfigured by the experiment service 404.

As noted above, the experiment service 404 may provide instructions tothe participating services 410 via the experiment clients 408 toconfigure the participating services 410 with particular levels ofparticular factors. Configuration of the participating services 410 maybe performed via an API. For example, upon generation of an electronicmarketing communication, the participating service generating theelectronic marketing communication may perform an API function call tonotify the experiment service 404 of the newly generated electronicmarketing communication. In some embodiments, the API function call mayinclude an identifier for the particular electronic marketingcommunication, an identifier for the intended recipient of theelectronic marketing communication, some other identifier that may beused by the experiment service 404 to determine if the electronicmarketing communication should be used as a treatment in an experiment.

Upon receiving the API function call from the experiment client 408, theexperiment service 404 may access a datastore containing one or moreexperiment models 406. The experiment service 404 may use the experimentmodels to determine whether the electronic marketing communication,consumer identifier, or other identifier indicated in the API functioncall is associated with an ongoing experiment. The experiment service404 may utilize the experiment models 406 to associate the electronicmarketing communication with an audience (or determine if the electronicmarketing communication is already associated with an audience,campaign, or the like), and to select a particular level for one or morefactors implemented by the calling participating service 410. Theselected particular level may be provided to the participating service410 in response to the API function call, and the participating service410 may use the assigned level to control generation of the electronicmarketing communication. Upon selecting a level for use by theparticipating service, the experiment service 404 may store anindication that the level is associated with a particular treatmentrepresented by the electronic marketing communication that is generatedby the participating service 410. It should also be appreciated that insome embodiments multiple participating services may participate in thegeneration of a single electronic marketing communication, such thateach participating service 410 performs a separate API function callrequesting levels from the experiment service 404. In such scenarios,the experiment service 404 may utilize an identifier associated with theconsumer recipient of the electronic marketing communication or a commonidentifier associated with the electronic marketing communication toselect which levels to apply to the electronic marketing communicationand to track the applied levels for use as a treatment. In this manner,the experiment service 404 may assign different levels to differentelectronic marketing communications to assist with audience selectionand to ensure that proper sampling and statistical analysis techniquesare employed (e.g., by ensuring a proper mix of levels acrosstreatments).

To track the results of the experiment, embodiments may receive userevents 414 and a tracking log 412 which are aggregated as a universalclickstream 416. The user events 414 may include, for example, userinterface interactions such as products or services viewed fromselections in emails, products or services viewed on a web page,products or services viewed in a mobile application, the amount of timespent viewing a particular product, advertisements clicked on,interactions with social media sites, or the like. The tracking log 412may include logs of events monitored by a promotion and marketingservice, such as product purchases, product shipments, and the like. Theuniversal clickstream 416 may represent an aggregation of the userevents 414, the tracking log 412, and any other data that may beemployed to determine the success of a particular experimentaltreatment.

Assessment of the results of the experiment may be performed byassociate data included in the universal clickstream 416 with particularexperiment treatments. As noted above, the universal clickstream 416includes data that may be analyzed to determine a performancecharacteristic of a given experimental treatment. This data may include,for instance, events that indicate that a consumer viewed a product orservice, that a consumer purchased a product or service, or variousother events. In some embodiments, experiment models may defineparticular events or criteria used to control measurement of performancecharacteristics derived from results. For example, a first experimentmay seek to analyze the impact of a particular factor on a productviewing rate, while another experiment may seek to analyze the impact ofa particular factor on a product purchase rate. Some experiments mayalso include analysis of multiple different results, such as experimentsthat seek to measure the impact of a particular factor on both a productviewing rate and a product purchase rate.

The universal clickstream 416 may be accessed by an initial dataaggregation service 420 and an enterprise data warehouse aggregationservice 422 to derive the results of running experiments. The initialdata aggregation service 420 may be used to generate initial real-timeresults for the purpose of test validation (e.g., verifying that levelsare properly being set for outgoing treatments), monitoring (e.g.,verifying that clickstream data associated with particular experimentsis being received), for quick turnaround experiments, and the like.

The enterprise data warehouse aggregation service 422 functions to storeand manage data derived from the received universal clickstream 416. Aspart of data warehousing operations, the enterprise data warehouseaggregation service 422 may apply an attribution model to attributeparticular events from the clickstream to particular consumers and toparticular electronic marketing communications. For example, theenterprise data warehouse aggregation service 422 may use a consumeridentifier associated with a particular consumer (e.g., as derived basedon a cookie or other tracking element associated with an entry in theclickstream), and identify the electronic marketing communications thathave been sent to that consumer. If a promotion associated with theevent in the universal clickstream was sent to the consumer previously,then the enterprise data warehouse aggregation service 422 may attributethe clickstream event (e.g., viewing the page for a promotion) with theelectronic marketing communication (e.g., an email mentioning theparticular promotion). In some embodiments, the enterprise datawarehouse aggregation service 422 may perform more complicated mappingsbetween the universal clickstream 416 and particular electronicmarketing communications. For example, a universal clickstream 416 eventmay only be associated with a particular electronic marketingcommunication if the clickstream event occurs within a particular timewindow (e.g., 1 hour, 1 day, 1 week, or one month) after transmission ofthe electronic marketing communication to the consumer. The attributionmodel 424 may include various rules and parameters for associationbetween particular electronic marketing communications and particularevents from the universal clickstream 416.

A data analysis service 428 may use data gathered by the initial dataaggregation service 420 and the enterprise data warehouse aggregationservice 422 to determine results of the experiment by determining theimpact of different treatments on the manner in which electronicmarketing communications are perceived by consumers. For example, thedata analysis service 428 may determine the correlations betweenparticular levels and a performance characteristic (e.g., the “success”)of one or more electronic marketing communications at increasing pageviews, product purchases, or the like. The data analysis service 428 mayalso perform regression analyses and other statistical modelingtechniques to identify the impact of particular levels on theperformance characteristics of an electronic marketing communication, acampaign, or the like. The data analysis service 428 may also performdifferent analyses based on the parameters of the experiment. Forexample, data associated with an “A/A” test experiment used to measurestatistical “noise” would be analyzed in a different manner than an“A/B” test used to compare two or more particular levels of a factor,which would in turn be analyzed differently from a “factorial”experiment designed to evaluate a plurality of factors and/or levels.

A data visualization service 426 may be employed to view the resultsderived from the initial data aggregation service 420 and the dataanalysis service 426. The data visualization service 426 may includevarious interfaces, tables, spreadsheets, and the like to provide usersof the system with the results of running experiments. The datavisualization service 426 may, for example, provide results in real-timefrom the initial data aggregation service 420 with more detailed dataprovided at regular intervals by the data analysis service 426.

The data analysis service 426 may also provide results to an experimentfeedback service 418. The experiment feedback service 418 may monitorresults provided by the data analysis service 426 and identifyparticular results that warrant adjustment to ongoing experiments orcreation of new experiments. For example, in the case of a “king of thehill” experiment that is designed to identify an optimal factor/levelpairing or group of factor/level pairings, if a particular treatmentperforms at least a threshold amount worse than one or more othertreatments (e.g., a certain number of standard deviations worse thanother treatments, or below some particular threshold performancecharacteristic), then the experiment feedback service 418 mayautomatically remove that treatment from use in future electronicmarketing communications by notifying the experiment service 404. Theexperiment feedback service 418 may include various additional andalternative rules, scripts, and criteria for altering runningexperiments and defining new experiments. For example, in someembodiments a script may implement the “king of the hill” experimenttype as described above with a lowest performing level of a particularfactor removed at periodic intervals until only a top level of thefactor remains.

Example Data Flow Among System Components

FIG. 5 illustrates an example data flow interaction 500 among componentsof a system implementing a system for conducting experiments usingelectronic marketing communications in accordance with some exemplaryembodiments of the present invention. In some embodiments, the data flow500 illustrates interaction among components of an apparatus such as theapparatus 200 described above.

The data flow 500 illustrates the interaction between an experimentservice 502, one or more communication services 504, and a responsetracking service 506. The experiment service 502 includes an experimentmanagement component 514, a communication service interface component510, an experiment configuration interface 512, a result analysiscomponent 522, and a result visualization component 524. Thecommunication service interface component 510 may receive and/or store aset of communication service capability data 516. The experimentmanagement component 514 may access a set of experiment model data 518.The audience management component 520, the communication services 504,and the response tracking component 506 may access a set of audiencedata 508.

The experiment configuration interface 512 may include one or moreinterfaces for receive manual or programmatic definition of experimentconfiguration parameters, such as described above with respect to theexperiment configuration circuitry described in FIG. 2. The experimentconfiguration interface 512 may provide the received experimentconfiguration parameters to the experiment management component 514 foruse in creating one or more experiments. The experiment managementcomponent 514 may be implemented, for example, by experiment managementcircuitry as described above with respect to FIG. 2.

The experiment management component 514 may interact with acommunication service interface component 510 to determine thecapabilities of the communication services 504. For example, thecommunication service interface component 504 may implement anapplication programming interface that provides a mechanism by which thecommunication services 504 may inform the experiment managementcomponent 514 of the particular factors and levels that may be used tocontrol each communication service 504. The associations betweenparticular communications services and available factors and/or levelsmay be stored in the set of communication service capability data 516.When the experiment management component 514 receives a set ofexperiment configuration parameters from the experiment configurationinterface 512, the experiment management component 514 may identify anyfactors and/or levels defined in the experiment configuration parametersand, using the communication service capability data 516, determinewhich of the communication services 504 support the factors and/orlevels defined in the experiment configuration parameters. At runtime,each communication service 504 may perform an API function call uponperforming an action for which the communication service 504 isconfigured to be controlled by a particular level provided by theexperiment management component 514. For example, upon reaching a pointin a process flow where a content selection service is prepared toselect content for an electronic marketing communication, the contentselection service may perform an API function call to receive one ormore levels from the experiment management component, such that thelevels impact the process by which content is selected for theelectronic marketing communication.

The experiment management component 514 may also interface with theaudience management component 520. The audience management component 520may provide an interface for selecting an audience for an experiment byaccessing a set of audience data 508. The audience data 508 may include,for example identifiers, contact information, and the like for one ormore consumers that are possible addressees of electronic marketingcommunications generated by embodiments of the present invention. Theaudience data 508 may be divided into various groupings, includingarbitrarily assigned groupings, groupings based on commoncharacteristics, or the like. In some embodiments, consumers arerandomly assigned into groups according to a hashing algorithm, andparticular experiments may be assigned to a particular group of groupsof the audience. In some embodiments, the hashing algorithm is saltedwith a particular value to provide for randomized assignment ofconsumers to particular groups. In some embodiments, the audiencemanagement component 520 provides for logging of these hashingoperations and the salt value applied to allow for mapping of particularaudiences to particular experiments even after the set of audience datais re-salted.

When receiving an API call from a communication service 504 via thecommunication service interface component 510, the experiment managementcomponent 512 may query the audience management component 520 and/orexperiment model data 518 to determine whether a consumer associatedwith the API call is part of a particular experiment. For example, theexperiment model data 518 may include a list of audience membersassociated with each particular experiment. A detailed example of adomain model for storing experiment data is described further below withrespect to FIG. 6.

The response tracking service 506 may monitor incoming data, such as auniversal clickstream as described above, to determine events that occurthat are possibly related to one or more running experiments. Theresponse tracking service 506 may provide these events to a resultanalysis component 522. The result analysis component 522 may beimplemented by result analysis circuitry, such as described above withrespect to FIG. 2. The result analysis component 522 may perform amapping between particular events and experimental treatments. Forexample, each event in the universal clickstream may be associated witha particular consumer, campaign, or the like. The result analysiscomponent 522 may analyze experiment model data 518 accessible to theexperiment management component 514 to determine whether any givenclickstream event is associated with a particular experiment. Forexample, the experiment model data 518 may include a mapping of eachconsumer to each experiment to which that consumer has been assigned. Ifthe consumer associated with the clickstream event is assigned to anexperiment, then the clickstream event may be analyzed to determine adata point for analysis for that particular experiment. A plurality ofresults, and/or data derived therefrom may be provided for viewing andanalysis via a result visualization component 524. The resultvisualization component 524 may display or otherwise provide one or moreinterfaces for selecting particular experiments, viewing the results ofthe selected experiment, and the like.

The result analysis component 522 may also compare the results of theparticular experiment to a set of criteria for altering an experiment,such as may be defined in a set of analysis scripts (not pictured). Theresult analysis component 522 may utilize these analysis scripts toprogrammatically alter running experiments and/or define newexperiments. For example, as indicated above, if a particular treatmentor level is associated with results that indicate the treatment or levelis performing particularly poorly, then the treatment or level may beremoved from future assignment while the experiment is still ongoing.

Each communication service 504 may include a communication managementcomponent 526 and an experiment interface component 528. Thecommunication management component 526 may include logic that enablesthe communication management component 526 to perform one or more partsof a process for generating an electronic marketing communication (e.g.,content selection, cadence selection, communication type selection,actual transmission of the electronic marketing communication, or thelike).

At particular points within the logic flow of the communicationmanagement component 526, the experiment interface component 528 may becalled to request experiment configuration parameters (e.g., levels)from the experiment management component 514. The experiment interfacecomponent 528 may be, for example, an API function executed by thecommunication service 504. The communication management component 526may provide an identifier, such as an electronic communicationidentifier or a consumer identifier, to the experiment interfacecomponent 528. In response, the communication management component 526may receive a level or levels as defined by the experiment interfacecomponent 528. These levels may be used by the communication managementcomponent 526 to alter the process by which the electronic marketingcommunication is created or transmitted by the communication service504. In this manner, embodiments advantageously allow for communicationservices to control how particular levels are implemented within thecommunication service, resulting in a loose coupling between theimplementation of the experiment service 502 and the communicationservices 504. Each communication service 504 may be expected to be ableto handle its own particular set of factors and levels, such that theonly input provided by the experiment service in response to a functioncall is one or more appropriate levels. The communication service 504may then utilize those levels to configure the outgoing electronicmarketing communication appropriately. The communication service 504 maynotify the experiment service 502 of which levels it is capable ofutilizing during runtime, upon initialization, or at various otherpoints in the process. In some embodiments, particular features of agiven communication service 504 may be dynamically enabled duringruntime of the experiment service 502 based on code updates applied tothe communication service 504 or according to various other criteria(e.g., a particular time/date passing, certain content being enabled forselection, new content being added to a database, or the like).

Exemplary Domain Model

FIG. 6 illustrates an exemplary domain model 600 for storing data (e.g.,an experiment model) associated with a particular experiment as used byan experiment service in accordance with some embodiments. It should beappreciated that while specific data structures and variables aredescribed with respect to the domain model 600, not all examples of suchdata structures will necessarily include the same structures and someembodiments may include additional structures or variables. The domainmodel 600 includes a data structure defining an experiment 602. Theexperiment structure 602 includes an identifier for the experiment, atype of the experiment (e.g., A/A test, A/B test, factorial test), adefined start time and end time, a text description of the experiment(e.g., a short description to indicate to a user the subject of theexperiment), the status of the experiment (e.g., pending, running,completed), and an audience salt value. Each experiment 602 may beassociated with a set of factors 604. The factors 604 may each include afactor identifier and a description of the factor. The factors 604 maybe compared to a set of communication service capability data todetermine which communication services support which particular factors,such that when a request is received from a communication service thatsupports a factor associated with the experiment, and appropriateresponse is provided. Each factor 604 may include two or more mutuallyexclusive levels 606. Each level 606 specifies a setting or settings fora communication service that is interpreted by the communication serviceto configure the communication service to act in a particular manner.The levels 606 are interpreted as settings 608 by the correspondingcommunication service. Each level may be a corresponding identifier anddescription. The levels 606 may include one or more settings 608 thatrepresent the application of the level to a particular communicationservice. The settings 608 may include a JavaScript Object Notation(JSON) map of the particular setting values as applied to theapplication. In some embodiments, an experiment management component mayrespond to an API request from a communication service with the JSON mapcorresponding to the settings for a particular level as determined forthat particular communication service.

Based on the experiment type, the experiment 602 includes a set of oneor more treatments 610, which are a combination of levels as derivedfrom the experiment factors. For example, an A/A test would typicallyinclude a single factor with a single level, repeated a number of timesacross different treatment groups. An A/B test would typically have asingle factor with two levels, resulting in two treatments. A factorialexperiment would typically include multiple factors, each with two ormore levels. The treatments 610 may therefore result from the standardfactorial combinations. Treatment names may be derived from particularexperiment configuration parameters, such as the experiment type, anexperiment ID, factor levels, and replicate number (e.g., in cases wherethe same treatment is sent multiple times). Treatment names maytherefore become an implementation detail that is also exposed totracking services, such as an enterprise data warehouse aggregationservice as described above. Treatment settings 612 may define the mannerin which the particular treatment is implemented by a givencommunication service.

The experiment 602 is associated with an audience definition 614. Theaudience definition 614 may be criteria-based, layer-bucket based, orsome other means, and may be open for extension. At runtime, theexperiment 602 assigns a treatment to a consumer upon identification ofthe consumer by a calling communication service. The results of thetreatment are subsequently tracked through a set of clickstream data asdescribed above. The audience definition 614 may include a plurality ofconsumers, represented as users 618. The audience definition 614 mayalso be associated with a group of treatments 616, which are made of aplurality of treatments 610 and which provide associations betweenparticular groups of treatments and groups of consumers 618.

When an experiment is defined by an experiment configuration component,various experiment configuration parameters may be utilized to generatethe experiment model pictured in FIG. 6. For example, the experimentconfiguration component may provide the capability to select a varietyof features and parameters that, in turn, are used to generate theexperiment model. The following tables depict some non-limiting examplesof features and configuration options that may be selected or enabledthrough the use of an experiment configuration component and whichinform creation of the experiment model 600.

TABLE 1 Feature Description Multi-service By applying the treatment atthe point of diversion, only users experiments (with exposed to theexperiment would be involved, so the treatment upstream filtering)groups can remain balanced. Simple A user may be assigned to multipleexperiments. Layers can simultaneous identify experiments that areassumed to be independent. In the experiments independence case, a usermay be tracked by all assigned experiments. Sophisticated In somescenarios, intermingling of experiments may cause data simultaneousanalysis problems. This feature provides for tracking of experimentsinterdependent experiments. Hold out group The ability to assign usersto experiment-level and longitudinal “hold-out” groups (longitudinal =across a set of experiments). Hold-out groups do not receive aparticular treatment, but are included in the analysis. Within-runreplicas Sometimes, it may be worthwhile to replicate an experimentwithin the same run (e.g., by having two treatments with the samelevels). An experiment may be set up with replica of the treatment, andthe experiment service may generate the appropriate replicas, assigningappropriate treatment groups to each. Back-to-back Sometimes it may beworthwhile to repeat an experiment during a replicas different timewindow. In this case, the experiment service may create an experimentclone, but link its analysis through a blocking factor. Audiences may bereshuffled between experiments with different salts. Exclude specific Insome cases, a particular combination of levels may produce a treatmentsknown-bad experience. In a do-no-damage mode, the experiment service mayallow a treatment tag that excludes it from assignment. By having allapplication settings visible for a treatment, the experiment service caninspect the settings for such conditions. Include more users Anexperimenter may want to add more users to some treatments in sometreatments than others. An experiment may configure its audiences withspecific ranges of audience members. These ranges may be non-contiguous. [0-10, 14-34, 500-562] Memory-free An experiment may assignusers randomly to audience groupings experiments at each run, thusgetting enough responses, but limiting multiple exposures for the samespecific user. Such a scenario may use a memory-free audience totreatment assignment strategy. For example, a performance test might usethis feature. Manual Dial- In the case of an experiment type, such asking-of-the-hill, an up/Dial-down experimenter may want to alter theproportion of users to favor ‘winning’ treatments over ‘losing’treatments. In this case, a user to treatment assignment strategy mayshift buckets from one treatment to another through the experimentservice update. Hold out groups may count as a bucket source or sink, ifan experiment wants to reduce or expand its reach. Automated Dial- Thesystem may perform audience grouping dynamically based on up/Dial downstatistical tests. Multi-touch Sometimes, an experiment has a firsttouch on one day, and then experiments multiple follow-up touches onsubsequent days. In this case, a consumer that gets assigned to such anexperiment would be set up to arrive at this experiment again. That is,any upstream filters may pass this user on subsequent days by queryingthe experiment service. In this case, the downstream service and theupstream service may both have settings in the treatment. Developer-onlyAn experiment may need to be tested before release to a real experimentsaudience. In this case, a treatment may include a list of ‘forced users’(e.g., QA testers, system developers) that always receive a treatmentwhen the developer-only setting is enabled. An experiment property mayinclude the developer-only experiments alongside production as well.Predicting Given a set of audience groupings and a number of treatments,and experiment reach some idea of filtering percent, this feature mayprovide a way to estimate the reach of each treatment, and thus thestatistical power. Audience filtering An audience criterion may specifysome derived segmentation, eligibility such as an engagement score(e.g., a frequency with which a given consumer has purchased a productor promotion, or a frequency with which the consumer views products orpromotions advertised in an electronic marketing communication). Someexperiments may test treatments against such segments. At treatmentassignment, an assignment strategy may determine the consumer's segmentand select a treatment based on that segment. Alternatively, the clientmay supply a segment along with the consumer, to determine a matchingtreatment. Stratified sub- If an audience criterion contains a certainproportion of rare but samples influential sub-groups, then treatmentgroups may also contain those same proportions of sub-groups. Anaudience service could provide this mapping. Preflight experiment Aheadof any experiment, a participating service may test a setting settingsto confirm its behavior. Then, that service may communicate the settingand values to the experimenter, who may include it in a level, withinthe experiment definition. Ability to apply Once the participatingservice has called the experiment client, it setting per user call mayapply the settings returned by the experiment client. These settings arecontrolled by the application, and may be defined ahead of anyexperiment. Attribution Once raw data arrives (e.g., via theclickstream), the raw data passes through aggregation and attribution.Depending on the experiment, different aggregation and attributionstrategies may be applied. Response funnel Embodiments may record allelements of a “push” platform response funnel (e.g., interactions with amobile application), including application opens, clicks, conversions,gross bookings, and/or gross revenue. Projection After analysis, somefactor levels may be identified as more important than others. Remainingfactor levels may be collapsed into a simpler predictive model. It maynot be known ahead of time which factors are important and which arenot. If the initial experiment is analyzed using a scripting tools, suchas R, then a data scientist may modify that script to include only theimportant effects and interactions. Covariate analysis After anexperiment ran, other attributes could be used to understand deeperrelationships. For example, consumer attributes may be used tounderstand how male/female gender, east/west coast locations, and thelike affect responses. Data transformation Sometimes the experimentermay want to understand average values, but other times, the experimentermay want to understand percentage lift, or other kinds of statistics.Based on the statistics, the experimenter may be able to use simpleaveraging, but other data may require log transformation, or othertransformations. While compiling responses, an experiment may specific adata aggregation pipeline from a small number of choices. Replay dataUnder some circumstances, it is required to replay raw data aggregationthrough the aggregation process. For example, an administrator may haverestored data, the data aggregation pipeline may have changed, outliersmay have been identified, or it may be desirable to remove a specificdate window (e.g., Black Friday). Outlier and date Once an experimenthas run, exclusion criteria may be specified window exclusion on theaudience, date range or treatment. These exclusions may be appliedduring data aggregation and subsequent analysis scripts. Automated Someexperiments can rely on results from previous experiments feedback intonew to down-select choices. In this case, an automated data aggregationexperiments pipeline followed by an analysis script may produce relativescores among treatments. Those scores may be used to set up a secondexperiment from the initial experiment, with reduced choices. Displayautomated The system may run the analysis scripts based on dataaggregation results output, and produce decoration data series forstatistical significance. These decoration data series could beincorporated into a dashboard Display adjusted The system may run adata-analyst modified analysis script and results display the results inthe same manner as the automated results.

Exemplary Processes for Implementing a Message Management System

FIGS. 7-11 illustrate flow diagrams depicting processes for implementingthe message management systems described above with respect to FIGS. 1-6to provide a dynamic experimentation service for use in generation ofelectronic marketing communications in accordance with some embodiments.

FIG. 7 illustrates a flow diagram depicting an example of a process 700for generating an electronic marketing communication for use in anexperiment in accordance with embodiments of the present invention. Theprocess illustrates how experiment configuration parameters may bereceived and used to define a group of recipients of particularexperiment treatments. As communication services generate and transmitelectronic marketing communications, the communication services interactwith an experiment service to “check in” and determine whether thecommunication service should alter its processing to participate in anexperiment. The process 700 therefore provides for altering an outgoingelectronic marketing communication to use that electronic marketingcommunication as an experiment treatment. The process 700 may beperformed by an apparatus, such as the apparatus 200 described abovewith respect to FIG. 2, and/or components of the experiment service asdescribed with respect to FIGS. 4-5.

At action 702, a set of experiment configuration parameters arereceived. As noted above, the experiment configuration parameters mayinclude, for example, particular factors and/or levels to be tested,particular results to be monitored (e.g., product purchase or productview), a type of experiment (e.g., A/A test, A/B test, factorial test),audience definitions, and various other flags or configuration settingsin support of the features identified above with respect to Table 1. Insome embodiments, the experiment configuration parameters may onlyidentify factors and not levels, and as such the experiment may cyclethrough all possible levels associated with a particular factor. Itshould also be appreciated that, prior to receiving the experimentconfiguration parameters, the process 700 may have initialized one ormore communication services which each provide an indication of whichfactors and/or levels the communication services are configured toutilize.

At action 704, an experiment model is generated based on the experimentconfiguration parameters. As described above with respect to FIG. 6, theexperiment model may include a defined set of factors, levels,treatments, settings, audience groupings, and the like that are designedto perform the experiment. At action 706, the experiment model is storedin a memory.

At action 708, a notification, function call, or the like is receivedfrom a communication service, indicating that the communication serviceis requesting a determination to be made as to whether to alter one ormore levels associated with generation or transmission of an electronicmarketing communication. The call may include a communication identifierindicating a particular outgoing electronic marketing communication(e.g., that is in the process of being generated), and/or an identifierfor a particular consumer. At action 710, a determination is made as towhether the communication identifier meets the requirements of theexperiment model. For example, the determination may indicate that aconsumer identified by the identifier is part of the audience of anexperiment, or that the particular electronic marketing communication isassociated with some other criteria that make the electronic marketingcommunication eligible for use in the experiment defined in theexperiment model.

At action 712, if the communication identifier is associated with theexperiment model, the process 700 proceeds to action 714. Otherwise, theprocess 700 proceeds to action 720. At action 714, the process 700selects an experiment level according to the experiment model. Forexample, the process 700 may select a particular treatment with one ormore levels for use by the calling communication service. Alternatively,in some embodiments the process 700 may not select any particular level,but may still associate the electronic marketing communication with theexperiment, such as in cases where a “control” or “hold out” group isspecified.

At action 716, the communication service is notified of the selectedlevel. In some embodiments, notification of the selected level mayinclude mapping the level to a particular setting or group of settingsassociated with application of the selected level to the particularapplication. At action 718, an association between the communicationidentifier and the particular level is stored for use in resultsanalysis to attribute a particular set of results with the particularfactor level. If the communication identifier is not identified asassociated with the experiment model, then the process 700 may end ataction 720 without providing any settings to the communication service,or by notifying the communication service that default processing shouldbe employed.

FIG. 8 illustrates a flow diagram depicting an example of a process 800for implementing a dynamic experimentation system from the perspectiveof a communication service in accordance with some embodiments. Asdescribed above, an experimentation service may serve to generateexperiment models and to handle calls from communication servicesrequesting levels to be used for generation or transmission ofelectronic marketing communications. The process 800 illustrates themanner in which these communication services may perform processing toapply factor levels to electronic marketing communications. The process800 may be performed by communication service circuitry, such as thecommunication service circuitry 216 described above with respect to FIG.2.

At action 802, the process determines a communication identifier for anoutgoing communication. The communication identifier may be a uniqueidentifier for a particular electronic marketing communication that hasbeen created by the communication service or another communicationservice, the communication identifier may be a unique identifier for anelectronic marketing communication under construction or to be generatedby the communication service, the communication identifier may be anidentifier associated with a proposed recipient of the electronicmarketing communication, or any combination thereof or incorporatingother mechanisms for identifying the particular electronic marketingcommunication. At action 804, a call is made to an experiment servicewith the communication identifier to request one or more levels to beused in generation of transmission of the electronic marketingcommunication. At action 806, a response is received from the experimentservice, indicating whether the communication identifier indicates thatthe electronic marketing communication is to have any levels applied. Ataction 808, a determination is made as to whether the response includesany levels and/or settings for use in generation or transmission of theelectronic marketing communication. If so, the process 800 proceeds toaction 810. If the response does not contain any levels or settings, orthe response includes an indication that no levels or settings are to beapplied, the process 800 proceeds to action 812.

At action 810, the electronic marketing communication is generated usingthe received experiment levels/settings. As described above, differentcommunication services may implement received experiment levels orsettings in different ways, depending upon the particular configurationof each communication service. For example, a content selection servicemay have specific content defined by the levels/settings, or the contentselection service may have a general category of content defined by thelevels/settings. An email cadence communication service may receivelevels/settings that alter the rate at which a particular consumerreceives email. In some embodiments, particular levels or settings mayalso have sub-levels that define subsets of broader categories ofsettings.

At action 812, if the response does not include particularlevels/settings, or if the response indicates no levels/settings will beprovided, the electronic marketing communication may be generatedaccording to default processing of the communication service.

FIG. 9 illustrates a flow diagram depicting an example of a process 900for attributing clickstream events to a particular electronic marketingcommunication in accordance with some embodiments. As described above,once an electronic marketing communication has been generated accordingto one or more levels specified by an experiment service, derivingresults from the experiment requires attributing clickstream events toparticular experiment treatments so that the impact of the differentlevels of the treatments can be assessed. The process 900 provides anexemplary method for providing this attribution. The process 900 may beperformed by result analysis circuitry, such as the result analysiscircuitry 214 described above with respect to FIG. 2.

At action 902, a clickstream event is received. The clickstream eventmay include one or more identifiers that identify the circumstancesunder which the clickstream event occurred. For example, theseidentifiers may include a tracking cookie or other identifier associatedwith a particular user, an identifier for a particular electronicmarketing communication that caused the clickstream event, or the like.At action 904, the identifier associated with the clickstream event maybe used to identify a particular electronic marketing communication orcommunications with which the identifier is associated. At action 906, adetermination is made as to whether the identifier is associated withany ongoing experiments. For example, if the clickstream event isassociated with a consumer that received an experimental treatmentrecently, then the clickstream event may be relevant to the experimentassociated with the experimental treatment received by the consumer. Ataction 908, the result associated with the clickstream event isidentified. As noted above, the result may be that the consumerpurchased a particular product or service, redeemed a discount orpromotion, viewed a product or service, or any other interaction asdescribed above. At action 910, the result is stored in association withthe particular experimental treatment. In some embodiments, such as inthe case where the identifier for the electronic marketing communicationindicates each of the levels of the experimental treatment, analysis maybe performed in real-time as soon as the association between theclickstream event and the treatment is identified. In other embodiments,the result and the communication are stored for later processing.

FIG. 10 illustrates a flow diagram depicting an example of a process1000 for dynamically altering an experiment in accordance with someexemplary embodiments of the present invention. As described above,embodiments may function to dynamically alter audiences, factors,levels, and treatments while experiments are running, such as in thecase where a particular level or treatment are providing poor results.In this regard, embodiments provide for the ability to utilize one ormore scripts that include a set of criteria for evaluating experimentresults and performing particular actions in response to particularcriteria occurring.

Scripts may be associated with particular experiments, or associatedwith more than one experiment. For example, a given script may indicatethat, if a treatment or a particular level is performing below athreshold level (e.g., below a particular number of standard deviationsfrom other levels, or below some absolute performance threshold), thenthat treatment should be discontinued for further use in the experiment.

Scripts may also be employed to dynamically define additional or futureexperiments. For example, a script may indicate that an experimentshould be run repeatedly, reducing the number of levels used bydiscarding the lowest performing level each time. As another example, ascript may examine several levels that each have sub-levels, and the topperforming level may be used as a basis for a further experimentexploring the different performance characteristics of the sub-levels ofthe top performing level. The process 1000 may be performed byexperiment management circuitry and/or experiment configurationcircuitry, such as the experiment management circuitry 212 and/orexperiment configuration circuitry 210 described above with respect toFIG. 2.

At action 1002, an experiment is initiated. As described above,initiation of an experiment may include generation of one or moretreatments for testing of different levels that are the subject of theexperiment. At action 1004, preliminary results from the experiment arereceived. For example, these preliminary results may be results receivedfrom an initial data aggregation service as described above with respectto FIG. 4. At action 1006, the preliminary results are mapped toparticular experiment treatments. At action 1008, a determination ismade as to whether the preliminary results meet a predefined threshold.If the results exceed the threshold, the process 1000 proceeds to action1012 to continue the experiment.

However, if the results are below the threshold, the treatmentcorresponding to the results that fail to meet the threshold is removedfrom the experiment at action 1010. In the present context, removal of atreatment may include eliminating the particular combination of levelsused to generate the treatment from being used in the generation ortransmission of additional electronic marketing communications byensuring that the particular level or combination of levels is no longerprovided to any communication services that call the experiment service.In this manner, the process 1000 advantageously provides for dynamicalteration of the experiment so ensure that the experiment “does noharm” by continuing to use a particular treatment that is exhibitingpoor results. Embodiments may also ensure that at least a minimum samplesize of results have been received for the treatment, to account forvariance in results, and various other programmatic rules may beemployed to modify ongoing experiments such as assisting with selectionof levels, removal of levels, addition of levels, adjusting of selectionweights for particular levels, combination of additional factors with aparticular level, or the like.

FIG. 11 illustrates a flow diagram depicting an example of a process1100 for managing an audience for an experiment in accordance with someembodiments. As more and more experiments are run, it becomesincreasingly likely that past runs of experiments will impact theresults of future experiments due to some consumers being subjected tomultiple different experimental treatments. To account for this effect,consumers may be periodically “reshuffled” in the audience toredistribute particular groups of consumers who have receivedexperimental treatments and thus reduce the impact of consumersreceiving multiple treatments in succession. However, the inventors haverecognized that reshuffling the audience into new groupings at arbitrarytimes can interfere with ongoing experiments and cause other problemswith audience blocking. Accordingly, the inventors have developedtechniques for tracking when such reshuffling events occur, along withwhich salt values are associated with which experiments in order toassist with later analysis of the results of the experiment. The process1100 may be performed by experiment management circuitry and/orexperiment configuration circuitry, such as the experiment managementcircuitry 212 and/or experiment configuration circuitry 210 describedabove with respect to FIG. 2.

At action 1102, an initial audience selection mapping is determined. Forexample, consumers may be apportioned into particular “buckets”, witheach bucket having a certain number of consumers associated with it.When performing experiments, a particular group of buckets may beassigned to the experiment, such that the consumers associated with theassigned buckets are assigned to the experiment.

At action 1104, one or more experiments are initiated using the initialmapping determined at action 1102. At action 1106, an instruction isreceived to reshuffle/re-salt/rehash the consumers into differentbuckets. At action 1108, the audience is reshuffled. At action 1110, thetime, circumstances, salt value, and other parameters associated withthe audience reshuffle are logged, such that the ongoing experiment canutilize the logged data to map the previous audience to a new audienceto account for factors such as the same consumer receiving the sametreatment multiple times.

As will be appreciated, computer program code and/or other instructionsmay be loaded onto a computer, processor or other programmableapparatus's circuitry to produce a machine, such that execution of thecode on the machine by the computer, processor, or other circuitrycreates the means for implementing various functions, including thosedescribed herein.

As described above and as will be appreciated based on this disclosure,embodiments of the present invention may be configured as methods,mobile devices, backend network devices, and the like. Accordingly,embodiments may comprise various means including entirely of hardware ora combination of software and hardware. Furthermore, embodiments maytake the form of a computer program product on at least onecomputer-readable storage medium having computer-readable programinstructions (e.g., computer software) embodied in the storage medium.Any suitable computer-readable storage medium may be utilized, includingnon-transitory hard disks, CD-ROMs, flash memory, optical storagedevices, magnetic storage devices, or the like.

Embodiments of the present invention have been described above withreference to block diagrams and flowchart illustrations of methods,apparatuses, systems and computer program products. It will beunderstood that each block of the circuit diagrams and processflowcharts, and combinations of blocks in the circuit diagrams andprocess flowcharts, respectively, can be implemented by various meansincluding computer program instructions. These computer programinstructions may be loaded onto a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the computer program product includes theinstructions which execute on the computer or other programmable dataprocessing apparatus create a means for implementing the functionsspecified in the flowchart block or blocks.

These computer program instructions may also be stored in acomputer-readable storage device that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablestorage device produce an article of manufacture includingcomputer-readable instructions for implementing the function discussedherein. The computer program instructions may also be loaded onto acomputer or other programmable data processing apparatus to cause aseries of operational steps to be performed on the computer or otherprogrammable apparatus, thereby producing a computer-implemented processsuch that the instructions executed on the computer or otherprogrammable apparatus cause performance of the steps and therebyimplement the functions discussed herein.

Accordingly, blocks of the block diagrams and flowchart illustrationssupport combinations of means for performing the specified functions,combinations of steps for performing the specified functions and programinstruction means for performing the specified functions. It will alsobe understood that each block of the circuit diagrams and processflowcharts, and combinations of blocks in the circuit diagrams andprocess flowcharts, can be implemented by special purpose hardware-basedcomputer systems that perform the specified functions or steps, orcombinations of special purpose hardware and computer instructions.

Many modifications and other embodiments of the inventions set forthherein will come to mind to one skilled in the art to which theseembodiments of the invention pertain having the benefit of the teachingspresented in the foregoing descriptions and the associated drawings.Therefore, it is to be understood that the embodiments of the inventionare not to be limited to the specific embodiments disclosed and thatmodifications and other embodiments are intended to be included withinthe scope of the appended claims. Although specific terms are employedherein, they are used in a generic and descriptive sense only and notfor purposes of limitation.

What is claimed:
 1. An apparatus for performing audience mapping for adynamic experimentation service, the apparatus comprising: experimentmanagement circuitry configured to: apportion each of a plurality ofconsumers to an associated audience group of a plurality of audiencegroups; determine a first audience mapping, the first audience mappingdefining a first relationship between each of the plurality of consumersand the associated audience group of the plurality of audience groups towhich each is apportioned; utilize a set of electronic communications,sent to a plurality of consumers, via a network, to perform one or morefirst experiments utilizing the first audience mapping, for eachexperiment of the one or more first experiments (1) assigning at leastone treatment from among a set of treatments to each electroniccommunication of the set of electronic communications, each electroniccommunication being transmitted to a consumer, wherein the set ofconsumers being selected from among the plurality of consumers making upa treatment group, wherein each experiment includes at least one factorand at least one level, repeated across a plurality of treatment groups;(2) receiving results of the at least one treatment from each consumer;(3) tracking results of each treatment; and (4) defining an audiencedefinition indicative of a group of treatments comprised of the set oftreatments and associations between particular treatments of the set oftreatments and the associated audience group; transmit the set ofelectronic communications; receive an instruction to re-apportion theplurality of consumers while the one or more first experiments are beingperformed and the results are still being received; re-apportion theplurality of consumers among the plurality of audience groups while theone or more first experiments are being performed and the results arestill being received; determine a second audience mapping while the oneor more first experiments are being performed and the results are stillbeing received, the second audience mapping defining a secondrelationship between each of the plurality of consumers and theassociated audience group of the plurality of audience groups to whicheach is re-apportioned, wherein the second relationship is differentfrom the first relationship; store log data indicative of a mappingbetween the first relationship and the second relationship, wherein themapping identifies movement of the plurality of consumers across theplurality of audience groups; provide access to the one or more firstexperiments, allowing utilization of the log data to map each of theplurality of consumers from one particular associated audience group toeach of the set of treatments; utilize the set of electroniccommunications, sent to the plurality of consumers, via the network, toperform one or more additional experiments utilizing the second audiencemapping while the one or more first experiments are being performed andthe results are still being received; and dynamically alter experimentalparameters while experiments are running based on the results stillbeing received in advance of further transmission of the electroniccommunications, wherein the experiment management circuitry comprises ascript to programmatically alter running experiments or define newexperiments based on the at least one treatment associated with theresults performing at least a threshold amount worse than one or moreother treatments and remove the at least one treatment from futureassignment while the experiments are still ongoing.
 2. The apparatus ofclaim 1, wherein experiment management circuitry performs the firstaudience mapping by a first hashing operation that generates a hashvalue for each of the plurality of consumers.
 3. The apparatus of claim2, wherein the first hashing operation uses a first salt value.
 4. Theapparatus of claim 3, wherein the second audience mapping is performedby a second hashing operation, and wherein the second hashing operationuses a second salt value different from the first salt value.
 5. Theapparatus of claim 4, wherein the experiment management circuitry isfurther configured to log the first salt value and the second salt valuewith the stored mapping.
 6. The apparatus of claim 1, wherein theexperiment management circuitry is further configured to provide themapping to result analysis circuitry for use in analysis of a set ofexperiment results.
 7. A method for performing audience mapping for adynamic experimentation service, the method comprising: determining afirst audience mapping using a processor, the first audience mappingdefining a first relationship between a plurality of consumers and theassociated audience group of the plurality of audience groups to whicheach is apportioned; utilize a set of electronic communications, sent toa plurality of consumers, via a network, for performing one or morefirst experiments utilizing the first audience mapping, for eachexperiment of the one or more first experiments, assigning at least onetreatment from among a set of treatments to each electroniccommunication of the set of electronic communications, each electroniccommunication being transmitted to a consumer, wherein the set ofconsumers is selected from among the plurality of consumers making upthe treatment group, wherein each experiment includes at least onefactor and at least one level, repeated across a plurality of treatmentgroups; receiving results of the at least one treatment from eachconsumer; tracking results of each treatment; and defining an audiencedefinition indicative of a group of treatments comprised of the set oftreatments and associations between particular treatments of the set oftreatments and the associated audience group; transmitting the set ofelectronic communications; receiving an instruction to re-apportion theplurality of consumers of consumers while the one or more firstexperiments are being performed and the results are still beingreceived; re-apportion the plurality of consumers among the plurality ofaudience groups of consumers while the one or more first experiments arebeing performed and the results are still being received; determining asecond audience mapping using the processor of consumers while the oneor more first experiments are being performed and the results are stillbeing received, the second audience mapping defining a secondrelationship between the plurality of consumers and the associatedaudience group of the plurality of audience groups to which each isre-apportioned, wherein the second relationship is different from thefirst relationship; storing log data indicative of a mapping between thefirst relationship and the second relationship, wherein the mappingidentifies movement of the plurality of consumers across the pluralityof audience groups; providing access to the one or more firstexperiments, allowing utilization of the log data to map each of theplurality of consumers from one particular associated audience group toeach of the set of treatments; utilizing the set of electroniccommunications, sent to the plurality of consumers, via the network, forperforming one or more additional experiments utilizing the secondaudience mapping while the one or more first experiments are beingperformed and the results are still being received; and dynamicallyaltering experimental parameters while experiments are running based onthe results still being received in advance of further transmission ofthe electronic communications, wherein the method executes a script toprogrammatically alter running experiments or define new experimentsbased on the at least one treatment associated with the resultsperforming at least a threshold amount worse than one or more othertreatments and remove the at least one treatment from future assignmentwhile the experiments are still ongoing.
 8. The method of claim 7,wherein the first audience mapping is performed by a first hashingoperation that generates a hash value for each of the plurality ofconsumers.
 9. The method of claim 8, wherein the first hashing operationuses a first salt value.
 10. The method of claim 9, wherein the secondaudience mapping is performed by a second hashing operation, and whereinthe second hashing operation uses a second salt value different from thefirst salt value.
 11. The method of claim 10, further comprising loggingthe first salt value and the second salt value with the stored mapping.12. The method of claim 7, further comprising using the mapping toanalyze a set of experiment results.
 13. A non-transitory computerreadable storage medium comprising instructions that, when executed by aprocessor, cause the processor to: apportion each of a plurality ofconsumers to an associated audience group of a plurality of audiencegroups; determine a first audience mapping, the first audience mappingdefining a first relationship between each of the plurality of consumersand the associated audience group of the plurality of audience groups towhich each is apportioned; utilize a set of electronic communications,sent to a plurality of consumers, via a network, to perform one or morefirst experiments utilizing the first audience mapping, for eachexperiment of the one or more first experiments (1) assigning at leastone treatment from among a set of treatments to each electroniccommunication of the set of electronic communications, each electroniccommunication being transmitted to a consumer, wherein the set ofconsumers being selected from among the plurality of consumers making upa treatment group, wherein each experiment includes at least one factorand at least one level, repeated across a plurality of treatment groups;(2) receiving results of the at least one treatment from each consumer;(3) tracking results of each treatment; and (4) defining an audiencedefinition indicative of a group of treatments comprised of the set oftreatments and associations between particular treatments of the set oftreatments and the associated audience group; transmit the set ofelectronic communications; receive an instruction to re-apportion theplurality of consumers while the one or more first experiments are beingperformed and the results are still being received; re-apportion theplurality of consumers among the plurality of audience groups while theone or more first experiments are being performed and the results arestill being received; determine a second audience mapping while the oneor more first experiments are being performed and the results are stillbeing received, the second audience mapping defining a secondrelationship between each of the plurality of consumers and theassociated audience group of the plurality of audience groups to whicheach is re-apportioned, wherein the second relationship is differentfrom the first relationship; store log data indicative of a mappingbetween the first relationship and the second relationship, wherein themapping identifies movement of the plurality of consumers across theplurality of audience groups; provide access to the one or more firstexperiments, allowing utilization of the log data to map each of theplurality of consumers from one particular associated audience group toeach of the set of treatments; utilize the set of electroniccommunications, sent to the plurality of consumers, via the network, toperform one or more additional experiments utilizing the second audiencemapping while the one or more first experiments are being performed andthe results are still being received; and dynamically alter experimentalparameters while experiments are running based on the results stillbeing received in advance of further transmission of the electroniccommunications, wherein the processor executes a script toprogrammatically alter running experiments or define new experimentsbased on the at least one treatment associated with the resultsperforming at least a threshold amount worse than one or more othertreatments and remove the at least one treatment from future assignmentwhile the experiments are still ongoing.
 14. The computer readablestorage medium of claim 13, wherein the first audience mapping isperformed by a first hashing operation that generates a hash value foreach of the plurality of consumers.
 15. The computer readable storagemedium of claim 14, wherein the first hashing operation uses a firstsalt value.
 16. The computer readable storage medium of claim 15,wherein the second audience mapping is performed by a second hashingoperation, and wherein the second hashing operation uses a second saltvalue different from the first salt value.
 17. The computer readablestorage medium of claim 16, wherein the instructions further cause theprocessor to log the first salt value and the second salt value with thestored mapping.
 18. The computer readable storage medium of claim 17,wherein the instructions further cause the processor to use the mappingto analyze a set of experiment results.